# Cycles — Complete Documentation

> Runtime authority for autonomous agents. Keep agents within approved spend, risk, and action limits. Open protocol, multi-language SDKs, Apache 2.0.

This file contains the full text of all Cycles documentation, optimized for LLM ingestion. For a lightweight navigation index, see llms.txt.

---

# Quickstart



# Architecture Overview: How Cycles Fits Together

Cycles is a runtime authority for autonomous agents. It sits between your application and the actions that cost money or carry risk.

This page describes the components, how they interact, and where each piece runs.

::: tip Prerequisites
This is a reference page. If you haven't set up Cycles yet, start with the [End-to-End Tutorial](/quickstart/end-to-end-tutorial) or [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack).
:::

## System overview

<ArchDiagramFull />

Your application talks to the **Cycles Server** (port 7878) at runtime. The **Cycles Admin Server** (port 7979) is the management plane where you create tenants, generate API keys, and configure budget ledgers. The **Cycles Events Service** is an outbound worker that delivers webhook notifications asynchronously and, when CyclesEvidence is enabled, signs evidence envelopes; its app port 7980 and management port 9980 should stay internal. All three services share the same Redis instance.

::: info Independent release cadences
Runtime, admin, events, and dashboard images ship patch releases independently. Current tagged versions are maintained in the [release matrix](/changelog#current-versions); use that matrix instead of assuming the four repositories share one patch number. Older admin servers that predate newer query parameters (e.g., `sort_by`, `search`) ignore them rather than erroring — the APIs follow an additive-parameter guarantee.
:::

## Components

### Cycles Protocol

The protocol specification defines the API contract. It is a language-agnostic OpenAPI 3.1 spec that any client or server can implement.

The protocol defines:

- Runtime endpoints for decisions, reservations, balances, event ingest, evidence retrieval, and signer JWKS publication
- The Subject hierarchy (tenant, workspace, app, workflow, agent, toolset)
- The reserve → execute → commit lifecycle
- Error codes and their semantics
- Idempotency guarantees
- Scope derivation rules

The spec lives at [cycles-protocol](https://github.com/runcycles/cycles-protocol).

### Cycles Server

The reference server implementation. It is a Spring Boot 3.5 application backed by Redis 7+.

**What it does:**

- Accepts HTTP requests from clients
- Validates API keys and enforces tenant isolation
- Executes atomic budget operations via Redis Lua scripts
- Maintains budget state (allocated, spent, reserved, debt)
- Runs a background expiry sweep to clean up abandoned reservations
- Computes CyclesEvidence content hashes synchronously, returns `cycles_evidence` refs when configured, and serves signed envelopes plus public signer JWKS

**Modules:**

| Module | Purpose |
|---|---|
| `cycles-protocol-service-api` | REST controllers, security filters, exception handling |
| `cycles-protocol-service-data` | Redis repository, Lua scripts, scope derivation, expiry service |
| `cycles-protocol-service-model` | Shared DTOs and enums |

**Why Redis and Lua:**

Budget enforcement under concurrency requires atomicity. A reservation must check and update multiple scope counters in a single operation. Redis Lua scripts execute atomically on the server, ensuring no race conditions between concurrent reservations.

Six Lua scripts handle the core operations:

| Script | Operation |
|---|---|
| `reserve.lua` | Check budgets across all scopes, reserve atomically |
| `commit.lua` | Record actual spend, release remainder, handle overage |
| `release.lua` | Return reserved budget to pool |
| `extend.lua` | Extend reservation TTL |
| `event.lua` | Record direct debit without reservation |
| `expire.lua` | Mark expired reservations and release their budget |

### Cycles Admin Server

The management plane for Cycles. It runs as a separate Spring Boot 3.5 service on port 7979 and shares the same Redis instance as the Cycles Server.

The optional [Cycles Admin Dashboard](/quickstart/deploying-the-cycles-dashboard) (Vue 3 SPA) sits in front of this server and exposes its operations as a web UI — useful for day-two ops without crafting curl commands.

**What it does:**

- Manages tenants (create, list, update, suspend, close)
- Creates and revokes API keys with granular permissions
- Creates budget ledgers and handles funding operations (credit, debit, reset, reset_spent, repay debt)
- Defines policies (caps, rate limits, TTL overrides) matched by scope patterns — **stored for future runtime enforcement; not yet evaluated by the Cycles Server in v0**
- Validates API keys (used by the Cycles Server for authentication)
- Maintains an audit log of all administrative operations

**Modules:**

| Module | Purpose |
|---|---|
| `cycles-admin-service-api` | REST controllers, auth interceptor, Spring Boot app |
| `cycles-admin-service-data` | Redis repositories, key service |
| `cycles-admin-service-model` | Shared domain models and DTOs |

**Authentication:** Cycles uses two auth schemes depending on the endpoint. `X-Admin-API-Key` is the bootstrap/operator key for tenant and key management, audit, admin-only budget operations, and a small runtime admin-on-behalf-of reservation surface. `X-Cycles-API-Key` is tenant-scoped and carries explicit permissions for budgets, policies, reservations, balances, events, and tenant self-service webhooks.

#### `X-Admin-API-Key` — bootstrap / system administration

Set via the `ADMIN_API_KEY` environment variable. Not scoped to any tenant.

| Endpoint | Method | Purpose |
|---|---|---|
| `/v1/admin/tenants/*` | POST, GET, PATCH | Create, list, update, suspend tenants |
| `/v1/admin/api-keys/*` | POST, GET, DELETE | Create, list, revoke API keys |
| `/v1/auth/validate` | POST | Validate an API key |
| `/v1/admin/audit/logs` | GET | Query audit logs |
| `/v1/admin/budgets` | PATCH | Update budget settings (overage policy, overdraft limit) — admin key only |
| `/v1/admin/budgets/freeze`, `/v1/admin/budgets/unfreeze` | POST | Admin-only budget state changes |
| `/v1/reservations`, `/v1/reservations/{id}`, `/v1/reservations/{id}/release` | GET / POST | Runtime admin-on-behalf-of inspection and force release |

#### `X-Cycles-API-Key` — tenant-scoped operations

Requires a key created via the admin API with the appropriate [permissions](/how-to/api-key-management-in-cycles#available-permissions).

| Endpoint | Method | Required Permission |
|---|---|---|
| `/v1/admin/budgets` | POST | `budgets:write` |
| `/v1/admin/budgets` | GET | `budgets:read` |
| `/v1/admin/budgets/fund` | POST | `budgets:write` |
| `/v1/admin/policies` | POST, PATCH | `policies:write` |
| `/v1/admin/policies` | GET | `policies:read` |
| `/v1/balances` | GET | `balances:read` |
| `/v1/reservations` | GET | `reservations:list` |
| `/v1/reservations` | POST | `reservations:create` |
| `/v1/reservations/{id}/commit` | POST | `reservations:commit` |
| `/v1/reservations/{id}/release` | POST | `reservations:release` |
| `/v1/reservations/{id}/extend` | POST | `reservations:extend` |
| `/v1/decide` | POST | *(valid key only)* |
| `/v1/events` | POST | *(valid key only)* |

PATCH `/v1/admin/budgets` (budget settings) is `X-Admin-API-Key`-only — it is not available to tenant keys. The legacy `admin:write` / `admin:read` permissions act as wildcards that satisfy any `*:write` / `*:read` requirement, but new keys should carry the granular permissions (`budgets:write`, `policies:write`, etc.) instead.

Note that the admin server enforces these per-endpoint permissions on the governance plane. The reference runtime server (port 7878) does **not** enforce per-endpoint permissions on the runtime plane — it checks key validity and tenant match only, so the `reservations:*` permission strings on runtime endpoints document the spec's intent rather than reference-server behavior.

::: tip Which header do I use?
If the endpoint manages **identity**, fleet-level audit, admin-only budget state, or operator force-release → `X-Admin-API-Key`.
If the endpoint is a tenant-scoped budget, policy, runtime, balance, or event call → `X-Cycles-API-Key`.
:::

#### Key provisioning

The two headers represent different keys with different lifecycles:

1. **`X-Admin-API-Key`** is a static secret you choose at deploy time. Set it as the `ADMIN_API_KEY` environment variable when starting the admin server. There is no API to create or rotate it — you manage it like any infrastructure secret (secrets manager, env vars, etc.).

2. **`X-Cycles-API-Key`** keys are created dynamically via `POST /v1/admin/api-keys` (authenticated with the admin key). Each key is scoped to one tenant and carries explicit permissions. The key secret (e.g., `cyc_live_abc123...`) is returned once at creation time.

**Bootstrap order:** deploy server with admin key → create tenant → create API key → use API key for budgets and runtime operations. See [Deploying the Full Cycles Stack](/quickstart/deploying-the-full-cycles-stack) for the step-by-step walkthrough.

**Why a separate server:**

Separating the management plane from the runtime enforcement plane lets you:

- Run the admin server in a restricted network (internal only) while the Cycles Server is accessible to applications
- Scale the enforcement server independently from the admin server
- Apply different access controls to management vs runtime operations

See the [Admin API reference](/admin-api/) for the full API, or the [governance spec](https://github.com/runcycles/cycles-protocol/blob/main/cycles-governance-admin-v0.1.25.yaml) for the authoritative OpenAPI definition.

### Cycles Events Service

The async webhook delivery and evidence signing service. It runs as a separate Spring Boot 3.5 service with an internal app port (7980) and management/actuator port (9980), and shares the same Redis instance.

**What it does:**

- Consumes delivery jobs from a Redis queue (`dispatch:pending`) via BLMOVE — a reliable-queue pattern that parks each claimed job on `dispatch:processing` until acknowledged, recovering orphans idle longer than `DISPATCH_PROCESSING_RECOVERY_IDLE_MS` (default 180000 ms)
- Delivers events to webhook endpoints via HTTP POST with HMAC-SHA256 signatures
- Retries failed deliveries with exponential backoff (configurable: default 5 retries, 1s–60s delay)
- Auto-disables subscriptions after consecutive failures (default threshold: 10)
- Expires stale deliveries after configurable max age (default: 24h)
- Cleans up expired ZSET index entries hourly
- Builds and Ed25519-signs CyclesEvidence envelopes when `EVIDENCE_SERVER_ID` and the signing key are configured
- Stores signed evidence envelopes content-addressed for the runtime server to serve at `GET /v1/evidence/{id}`

**Why a separate service:**

| Concern | Admin Server | Events Service |
|---------|-------------|----------------|
| Workload | Synchronous CRUD, operator-facing | Asynchronous delivery and signing, variable latency |
| Scaling | Scale with admin traffic | Scale with webhook volume |
| Failure isolation | Admin stays responsive during delivery backlog | Delivery retries don't block admin API |
| Concurrency | Single instance | Multiple instances safe (the BLMOVE claim is atomic) |

**Optional:** If the events service is not deployed, admin and runtime servers operate normally. Webhook events and deliveries accumulate in Redis (bounded by TTL) and are processed when the events service starts. If CyclesEvidence is configured on the runtime server but the events service is down, responses may carry `cycles_evidence` refs while `GET /v1/evidence/{id}` returns transient `404` until the signer catches up.

See [Deploying the Events Service](/quickstart/deploying-the-events-service) for setup, [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol) for webhook delivery, and [CyclesEvidence Envelopes](/protocol/cycles-evidence-envelopes-in-cycles) for evidence signing and verification.

### Cycles MCP Server

A [Model Context Protocol](https://modelcontextprotocol.io) server that exposes Cycles runtime authority as MCP tools. MCP-compatible AI hosts (Claude Desktop, Claude Code, Cursor, Windsurf) discover and call these tools automatically.

**What it does:**

- Exposes 9 MCP tools covering the full Cycles protocol (reserve, commit, release, extend, decide, balance, events, reservations)
- Ships 3 built-in prompts for integration code generation, budget debugging, and strategy design
- Provides resources for inspecting balances and reservation state
- Wraps the `runcycles` TypeScript client internally — talks to the Cycles Server via HTTP

**When to use it:**

Use the MCP server when your agent host supports MCP and you want to expose Cycles tools without an SDK integration. Adding it to the tool configuration is sufficient for discovery, not for hard enforcement: the standalone server does not wrap the host's other actions. Add **Cycles Budget Guard for Claude Code** or a mandatory handler, gateway, harness, or service boundary when the action must not bypass Cycles. See [Getting Started with the MCP Server](/quickstart/getting-started-with-the-mcp-server).

### Cycles Spring Boot Starter

A client library that integrates Cycles into Spring Boot applications. It provides two usage modes:

1. **Declarative** — The `@Cycles` annotation wraps methods in a reserve → execute → commit lifecycle automatically via Spring AOP
2. **Programmatic** — The `CyclesClient` interface can be injected and used directly for fine-grained control

**Key components:**

| Component | Purpose |
|---|---|
| `@Cycles` annotation | Declarative budget enforcement on methods |
| `CyclesAspect` | AOP interceptor that drives the lifecycle |
| `CyclesLifecycleService` | Orchestrates reserve/execute/commit/release |
| `CyclesClient` / `DefaultCyclesClient` | HTTP client using Spring WebClient |
| `CyclesContextHolder` | ThreadLocal access to reservation state mid-execution |
| `CyclesExpressionEvaluator` | SpEL evaluation for dynamic estimates and actuals |
| `CyclesFieldResolver` | Interface for dynamic Subject field resolution |
| `CommitRetryEngine` | Same-key settlement retry, durable journal replay, and expired-commit event fallback |
| `CyclesProperties` | Spring Boot configuration properties |

## Request flow

Here is what happens when an `@Cycles`-annotated method is called:

### 1. Estimate evaluation

The SpEL expression in the annotation is evaluated against method parameters to produce a numeric estimate.

### 2. Reservation request

The starter sends `POST /v1/reservations` to the Cycles server with the Subject, Action, estimate, TTL, and overage policy.

### 3. Atomic budget check (server side)

The server derives all affected scopes from the Subject, then executes `reserve.lua`. The Lua script:

- Checks each scope has sufficient remaining budget (`allocated - spent - reserved - debt >= estimate`)
- Checks no scope has outstanding debt or is over-limit
- If all checks pass, atomically increments the `reserved` counter on every scope
- Stores the reservation record with its TTL

### 4. Decision returned

A successful live reservation returns `ALLOW` or `ALLOW_WITH_CAPS`. If a live reservation is denied, the server returns a protocol error such as `409 BUDGET_EXCEEDED`; `decision: DENY` is reserved for `/v1/decide` and dry-run reservation evaluations.

### 5. Method execution

If allowed, the starter runs the annotated method. During execution:

- A heartbeat thread periodically extends the reservation TTL
- The method can access `CyclesContextHolder` to read caps or set metrics

### 6. Commit

After the method returns, the starter evaluates the `actual` expression, durably journals the settlement, and sends `POST /v1/reservations/{id}/commit`. The server executes `commit.lua` to record actual spend and release the unused remainder. Ambiguous outcomes retain the same-key record for replay; an expired commit switches to a direct event.

### 7. Error path

If the method throws, the starter sends `POST /v1/reservations/{id}/release` to return all reserved budget to the pool.

## Data model

All budget state lives in Redis. The key concepts:

### Scopes

A scope is a budgeting boundary derived from the Subject hierarchy. A single reservation may affect multiple scopes. For example, a reservation with `tenant=acme, workspace=prod, app=chatbot` affects three scopes:

- `tenant:acme`
- `tenant:acme/workspace:prod`
- `tenant:acme/workspace:prod/app:chatbot`

### Balances

Each scope tracks:

| Field | Meaning |
|---|---|
| `allocated` | Total budget assigned to this scope |
| `spent` | Committed actual usage |
| `reserved` | Currently held by active reservations |
| `remaining` | `allocated - spent - reserved - debt` |
| `debt` | Negative balance from overdraft commits |
| `overdraft_limit` | Maximum allowed debt |
| `is_over_limit` | Whether `debt > overdraft_limit` |

### Reservations

Each reservation is stored with:

- Unique ID
- Subject and action metadata
- Reserved amount and unit
- Status (ACTIVE, COMMITTED, RELEASED, EXPIRED)
- TTL and grace period timestamps
- Idempotency key and payload hash

## Authentication

Budget and reservation requests authenticate via the `X-Cycles-API-Key` header. Each API key is associated with a tenant. The server enforces that `subject.tenant` matches the key's tenant — a key for tenant A cannot create reservations for tenant B.

Public runtime endpoints are intentionally narrow: the `/actuator/health/liveness` and `/actuator/health/readiness` probes, `GET /v1/evidence/{id}` (a content-addressed capability URL), and `GET /v1/.well-known/cycles-jwks.json` (public verification keys only). Since cycles-server 0.1.25.45 the aggregate `/actuator/health`, `/actuator/prometheus`, and the OpenAPI/Swagger docs paths require the `X-Admin-API-Key` header. Evidence retrieval does not expose ledger state; the unguessable `evidence_id` is the lookup capability.

Since 0.1.25.46 the public evidence and JWKS endpoints are rate-limited per client IP — 300 requests/minute by default (on by default; `cycles.public-rate-limit.*` properties). Exceeding the window returns `429` with `error=LIMIT_EXCEEDED` and a `Retry-After` header.

## Deployment topology

A typical deployment:

<DeploymentDiagram />

Multiple Cycles server instances can run behind a load balancer. All state is in Redis, so the server is stateless. The admin server is typically on an internal network, accessible only to operators and CI/CD pipelines. The events service is optional — if deployed, it consumes delivery jobs from Redis, delivers webhooks with HMAC-SHA256 signatures, and signs CyclesEvidence envelopes when the shared evidence identity is configured.

Non-Spring clients (Python, TypeScript/Node.js, Go) can use the protocol directly via HTTP — the client libraries are convenience layers, not a requirement. MCP-compatible agents (Claude Desktop, Claude Code, Cursor, Windsurf) can use the Cycles MCP Server for a zero-code integration path.

## Next steps

- [Tenants, Scopes, and Budgets](/how-to/understanding-tenants-scopes-and-budgets-in-cycles) — how tenants, scopes, and budgets work together as a unified model
- [Deploying the Full Cycles Stack](/quickstart/deploying-the-full-cycles-stack) — zero to working deployment with all components
- [Self-Hosting the Cycles Server](/quickstart/self-hosting-the-cycles-server) — server-specific configuration and deployment
- [API Reference](/api/) — interactive endpoint documentation
- [Getting Started with the MCP Server](/quickstart/getting-started-with-the-mcp-server) — add runtime authority to Claude Desktop, Claude Code, Cursor, or Windsurf
- [Getting Started with the Python Client](/quickstart/getting-started-with-the-python-client) — integrate with your Python app
- [Getting Started with the TypeScript Client](/quickstart/getting-started-with-the-typescript-client) — integrate with your TypeScript/Node.js app
- [Getting Started with the Spring Boot Starter](/quickstart/getting-started-with-the-cycles-spring-boot-starter) — integrate with your Spring app



# Deploy the Cycles Admin Dashboard

The Cycles Admin Dashboard is a Vue 3 single-page app that sits in front of [`cycles-server-admin`](https://github.com/runcycles/cycles-server-admin) and provides an operations-oriented UI for tenants, budgets, webhooks, and incident response. It's a thin client — all state lives in the admin server; the dashboard just visualises it and calls admin API endpoints on your behalf.

<p align="center">
  <img src="/images/cycles-dashboard-demo.gif" alt="End-to-end walkthrough of the main operator flows — tenants, budgets, reservations, webhooks" width="800" loading="lazy"/>
  <br/>
  <em>End-to-end walkthrough of the main operator flows</em>
</p>

::: info When to deploy the dashboard
If you only need SDK integration (Python, TypeScript, Spring, Rust, MCP), you can skip this page — the dashboard is optional. Deploy it when you want a UI for day-two operations: investigating events, freezing a runaway budget, rotating API keys, replaying missed webhooks, or force-releasing hung reservations during an incident.
:::

## What you get

A single pane for all operator tasks, capability-gated by the admin key:

| Page | Purpose |
|---|---|
| **Overview** | Single-request aggregated health dashboard — entity counts, top offenders, failing webhooks, over-limit scopes |
| **Tenants** | Tenant list + detail with budgets, API keys, and policies tabs |
| **Budgets** | Tenant-scoped budget list with utilization/debt bars; inline `RESET` and `RESET_SPENT` |
| **Events** | Correlation-first investigation tool with expandable detail rows |
| **API Keys** | Cross-tenant key list with masked IDs, permissions, status filters |
| **Webhooks** | Subscription health (green/yellow/red) + delivery history, replay, test |
| **Reservations** | Hung-reservation force-release during incident response (runtime-plane admin-on-behalf-of), with "View evidence" links when the runtime server emits evidence refs |
| **Evidence** | Retrieve and inspect a signed evidence envelope by id; resolve the signer against the runtime server's published JWK Set |
| **Audit** | Compliance query tool with CSV/JSON export |

Incident-response actions (freeze budget, suspend tenant, revoke API key, pause webhook, force-release reservation, emergency tenant-wide freeze) are one-click with confirmation and blast-radius summaries.

### Power-user features (v0.1.25.24+)

- **Command palette** — press `Cmd+K` on macOS or `Ctrl+K` on Linux/Windows to jump to any tenant, budget, webhook, or API key by ID or name. The palette also exposes the common incident actions without navigating.
- **Bulk action lanes** — the Tenants and Webhooks pages expose a filter-then-bulk workflow that maps onto the admin bulk endpoints (`POST /v1/admin/tenants/bulk-action`, `POST /v1/admin/webhooks/bulk-action`). The UI surfaces the `expected_count` safety gate, the 500-match `LIMIT_EXCEEDED` ceiling, and per-row succeeded/failed/skipped results.
- **Tenant hierarchy breadcrumbs** — detail pages show `tenant → workspace → app → workflow` so operators can navigate back up the scope path without losing context.
- **RESET_SPENT inline funding** — the Budgets page exposes `RESET_SPENT` alongside `RESET`, clearing `spent` for a new billing period without disturbing `reserved`/`debt`. Matches the server-side funding operation added in `cycles-server-admin` v0.1.25.18.

## Architecture

The dashboard is a static SPA served by nginx. It talks to **two backends** — the governance plane (`cycles-server-admin`) for tenants, budgets, policies, webhooks, events, and audit; and the runtime plane (`cycles-server`) for reservation force-release during incident response. Both are reverse-proxied through the dashboard's own nginx so the browser sees everything as same-origin and CORS is not involved in a standard production deployment.

<DashboardArchDiagram />

The default path (left branch) carries the governance/admin pages. The split path (right branch) carries runtime-plane Reservations and Evidence calls, including force-release, evidence lookup, and signer JWKS resolution.

The nginx routing split in `default.conf.template` (the TLS variant is `nginx-ssl.conf.example`):

| Request path | Upstream | Used by |
|---|---|---|
| `/v1/reservations*`, `/v1/evidence*`, `/v1/.well-known/cycles-jwks.json` | `cycles-server:7878` | Reservations and Evidence pages — force-release, envelope lookup, signer-key resolution |
| `/v1/*` (everything else) | `cycles-admin:7979` | All other dashboard pages |

Both backends authenticate the same `X-Admin-API-Key` header. On the runtime plane, force-release is an admin-on-behalf-of call — the runtime server validates the admin key and records the actor in the audit trail.

## Quick start (development)

For local development against a running admin server:

```bash
git clone https://github.com/runcycles/cycles-dashboard.git
cd cycles-dashboard
npm install
npm run dev
```

Dashboard opens at `http://localhost:5173`. The Vite dev server mirrors the production routing split:

- `/v1/reservations*`, `/v1/evidence*`, `/v1/.well-known/cycles-jwks.json` → `localhost:7878` (`cycles-server` — runtime plane)
- `/v1/*` (everything else) → `localhost:7979` (`cycles-admin` — governance plane)

**You need both backends running.** See [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack) to bring them up — or `cd ../cycles-server-admin && ADMIN_API_KEY=your-key docker compose up -d` if you only want admin + Redis and plan to run `cycles-server` separately. Log in with the same `ADMIN_API_KEY` you set on the servers (both admin and runtime must share the same key for force-release to work).

::: tip CORS for dev mode
If the admin or runtime server rejects your dev browser with a CORS error, set `DASHBOARD_CORS_ORIGIN=http://localhost:5173` on **both** containers. (In production with the nginx reverse proxy, CORS is not needed — the dashboard's nginx makes every backend call same-origin.)
:::

## Production (Docker + Caddy)

Recommended production setup uses Caddy for automatic HTTPS. Only ports 443 and 80 are exposed; admin server and Redis stay on the internal Docker network.

```yaml
# docker-compose.prod.yml
services:
  caddy:
    image: caddy:2-alpine
    restart: unless-stopped
    ports:
      - "443:443"
      - "80:80"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy-data:/data
    depends_on:
      - dashboard
    networks:
      - cycles

  dashboard:
    image: ghcr.io/runcycles/cycles-dashboard:0.1.25.85
    restart: unless-stopped
    # No exposed ports — only reachable through Caddy.
    environment:
      ADMIN_UPSTREAM: ${ADMIN_UPSTREAM:-http://cycles-admin:7979}
      RUNTIME_UPSTREAM: ${RUNTIME_UPSTREAM:-http://cycles-server:7878}
    depends_on:
      cycles-admin:
        condition: service_healthy
      cycles-server:
        condition: service_healthy
    networks:
      - cycles

  # Runtime plane — reservation force-release, evidence lookup, and signer JWKS
  # go here via /v1/reservations*, /v1/evidence*, and
  # /v1/.well-known/cycles-jwks.json.
  # Its ADMIN_API_KEY must match cycles-admin's so admin-on-behalf-of calls
  # authenticate on both sides.
  cycles-server:
    image: ghcr.io/runcycles/cycles-server:0.1.25.59
    restart: unless-stopped
    environment:
      REDIS_HOST: redis
      REDIS_PORT: 6379
      REDIS_PASSWORD: ${REDIS_PASSWORD:?REDIS_PASSWORD must be set}
      ADMIN_API_KEY: ${ADMIN_API_KEY:?ADMIN_API_KEY must be set}
      WEBHOOK_SECRET_ENCRYPTION_KEY: ${WEBHOOK_SECRET_ENCRYPTION_KEY:?WEBHOOK_SECRET_ENCRYPTION_KEY must be set}
      DASHBOARD_CORS_ORIGIN: ${DASHBOARD_ORIGIN:-https://admin.example.com}
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:7878/actuator/health/readiness"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s
    depends_on:
      redis:
        condition: service_healthy
    networks:
      - cycles

  # Governance plane — tenants, budgets, policies, webhooks, events, audit.
  cycles-admin:
    image: ghcr.io/runcycles/cycles-server-admin:0.1.25.55
    restart: unless-stopped
    environment:
      REDIS_HOST: redis
      REDIS_PORT: 6379
      REDIS_PASSWORD: ${REDIS_PASSWORD:?REDIS_PASSWORD must be set}
      ADMIN_API_KEY: ${ADMIN_API_KEY:?ADMIN_API_KEY must be set}
      WEBHOOK_SECRET_ENCRYPTION_KEY: ${WEBHOOK_SECRET_ENCRYPTION_KEY:?WEBHOOK_SECRET_ENCRYPTION_KEY must be set}
      WEBHOOK_SECRET_ALLOW_PLAINTEXT: "false"
      DASHBOARD_CORS_ORIGIN: ${DASHBOARD_ORIGIN:-https://admin.example.com}
      EVENT_TTL_DAYS: ${EVENT_TTL_DAYS:-90}
      DELIVERY_TTL_DAYS: ${DELIVERY_TTL_DAYS:-14}
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:7979/actuator/health/readiness"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s
    depends_on:
      redis:
        condition: service_healthy
    networks:
      - cycles

  # Async webhook delivery and optional CyclesEvidence signing.
  cycles-events:
    image: ghcr.io/runcycles/cycles-server-events:0.1.25.25
    restart: unless-stopped
    environment:
      REDIS_HOST: redis
      REDIS_PORT: 6379
      REDIS_PASSWORD: ${REDIS_PASSWORD:?REDIS_PASSWORD must be set}
      WEBHOOK_SECRET_ENCRYPTION_KEY: ${WEBHOOK_SECRET_ENCRYPTION_KEY:?WEBHOOK_SECRET_ENCRYPTION_KEY must be set}
      # Evidence signing is off unless these are set consistently with cycles-server:
      # EVIDENCE_SERVER_ID, EVIDENCE_SIGNING_SIGNER_DID,
      # EVIDENCE_SIGNING_PRIVATE_KEY_HEX.
    depends_on:
      redis:
        condition: service_healthy
    networks:
      - cycles

  redis:
    image: redis:7-alpine
    restart: unless-stopped
    environment:
      REDIS_PASSWORD: ${REDIS_PASSWORD:?REDIS_PASSWORD must be set}
    command: ["sh", "-c", "redis-server --appendonly yes --requirepass \"$${REDIS_PASSWORD}\""]
    volumes:
      - redis-data:/data
    healthcheck:
      test: ["CMD-SHELL", "redis-cli -a \"$${REDIS_PASSWORD}\" ping"]
      interval: 5s
      timeout: 3s
      retries: 5
    networks:
      - cycles

volumes:
  redis-data:
  caddy-data:

networks:
  cycles:
```

::: warning Both backends required
The dashboard's nginx routes `/v1/reservations*`, `/v1/evidence*`, and `/v1/.well-known/cycles-jwks.json` to `cycles-server:7878`. If you omit `cycles-server`, every governance page works but Reservations and Evidence fail with a 502. The `ADMIN_API_KEY` **must be the same** on both `cycles-admin` and `cycles-server` — admin-on-behalf-of force-release calls authenticate on both sides. Signed evidence also requires `cycles-server-events` to run with the matching evidence private key.
:::

```
# Caddyfile
admin.example.com {
    reverse_proxy dashboard:80
}
```

Deploy:

```bash
# Generate the secrets into shell variables first. Do NOT write the
# $(openssl ...) calls inside a quoted heredoc — the command substitution
# would never run, and every deployment would use the literal string
# "$(openssl rand -base64 32)" as its key.
ADMIN_API_KEY="$(openssl rand -base64 32)"
REDIS_PASSWORD="$(openssl rand -base64 32)"
WEBHOOK_SECRET_ENCRYPTION_KEY="$(openssl rand -base64 32)"

cat > .env <<EOF
ADMIN_API_KEY=$ADMIN_API_KEY
REDIS_PASSWORD=$REDIS_PASSWORD
WEBHOOK_SECRET_ENCRYPTION_KEY=$WEBHOOK_SECRET_ENCRYPTION_KEY
EOF

docker compose -f docker-compose.prod.yml up -d
```

Open `https://admin.example.com`, log in with the generated `ADMIN_API_KEY`.

::: warning Admin API key is the only credential
The dashboard uses `X-Admin-API-Key` exclusively for authentication. There is no user login, no SSO out of the box. **Treat the admin key as a root credential** — rotate it regularly, store it in a secrets manager, and consider placing the dashboard behind SSO or VPN for defense in depth.
:::

## Authentication and session handling

1. User enters the admin API key on the login page.
2. Dashboard calls `GET /v1/auth/introspect` to validate and retrieve capabilities.
3. Sidebar navigation is gated by capability booleans (`view_overview`, `view_budgets`, etc.) — the admin server decides what's visible.
4. On 401/403 from any API call, the session is cleared and the user is redirected to login.
5. Key is stored in `sessionStorage` — survives page refresh, cleared on tab/browser close. Never written to `localStorage` or cookies.
6. Session idle timeout (30 min) and absolute timeout (8 h) enforced client-side, checked every 15 s.
7. Login rate limiting — exponential backoff after 3 failed attempts (5 s → 10 s → 20 s → 40 s → 60 s cap).

## Polling cadence

Each page manages its own polling lifecycle via a `usePolling` composable. Intervals are tuned per page — most default to 60 s; Events polls more aggressively, Audit not at all.

| Page | Interval | Behaviour |
|---|---|---|
| Overview | 30 s | Pause on tab hidden, 2× backoff on error (max 5 min) |
| Events | 15 s | Same |
| Budgets | 60 s | Same |
| Webhooks | 60 s | Same |
| Tenants | 60 s | Same |
| Audit | Manual only | Explicit "Run Query" button |

## Hardening checklist

- **Never expose ports 7878, 7979, or 6379** to the public internet. Only the TLS proxy (443/80) should be reachable; the runtime, admin, and Redis containers must stay on the internal Docker network.
- **HTTPS only** — the admin key is sent on every request. Caddy or a load balancer with TLS 1.2+ in front.
- **Rotate the admin key** periodically (`openssl rand -base64 32`).
- **Set `REDIS_PASSWORD`** — the default has no authentication.
- **Store secrets in a secrets manager** (Vault, AWS Secrets Manager, etc.), not in git or `.env` files committed to the repo.
- **Subresource Integrity (SRI)** hashes are baked into all production assets via `vite-plugin-sri-gen`.
- The default `default.conf.template` ships with `X-Frame-Options: DENY`, `X-Content-Type-Options: nosniff`, CSP, Permissions-Policy, and `server_tokens off`. The TLS variant (`nginx-ssl.conf.example`) adds HSTS.

## Monitoring

- Admin server exposes `/actuator/health/liveness` and `/actuator/health/readiness` — use those for probes. The aggregate `/actuator/health` (like `/actuator/prometheus`) requires the `X-Admin-API-Key` header.
- `GET /v1/admin/overview` is a good synthetic monitoring target — if it returns 200, the full stack (Redis + admin + auth) is working.
- Alert on the overview payload's `failing_webhooks` and `over_limit_scopes` arrays.
- For deeper metric-based alerting, see [Observability Setup](/how-to/observability-setup) and [Monitoring and Alerting](/how-to/monitoring-and-alerting).

## Environment variables

| Variable | Required | Default | Description |
|---|---|---|---|
| `ADMIN_API_KEY` | Yes | — | Admin API key for `X-Admin-API-Key`. **Must be the same value on `cycles-admin` and `cycles-server`** — admin-on-behalf-of calls (force-release reservation) authenticate on both. |
| `REDIS_PASSWORD` | Recommended | (empty) | Redis authentication password |
| `WEBHOOK_SECRET_ENCRYPTION_KEY` | Yes | — | AES-256-GCM key for webhook signing secrets at rest; required by the admin and events services unless their explicit local-development plaintext escape hatch is enabled |
| `DASHBOARD_CORS_ORIGIN` | Dev only | `http://localhost:5173` | CORS origin — only needed when the browser calls the admin server directly (dev mode); unused in standard production |
| `ADMIN_UPSTREAM` | No | `http://cycles-admin:7979` | Governance-plane upstream for the dashboard container's nginx proxy |
| `RUNTIME_UPSTREAM` | No | `http://cycles-server:7878` | Runtime-plane upstream for reservations, evidence lookup, and signer JWKS |
| `EVIDENCE_SERVER_ID` / `EVIDENCE_SIGNING_SIGNER_DID` | Evidence only | (empty) | Backend identity values required on both `cycles-server` and `cycles-events` for signed evidence |
| `EVIDENCE_SIGNING_PRIVATE_KEY_HEX` | Evidence only | (empty) | Backend private Ed25519 signing seed. Set only on `cycles-events`; never set on the dashboard or runtime server |
| `EVIDENCE_SIGNING_KID` | Evidence JWKS only | derived | Public JWK `kid` label for the runtime server's signer JWKS. Not key material and not read by `cycles-events` |

The dashboard itself has no application-level server configuration — it's a static SPA. The backend URLs are set by the proxy:
- **Development:** Vite proxy in `vite.config.ts` (defaults: `localhost:7979` admin, `localhost:7878` runtime).
- **Production:** `ADMIN_UPSTREAM` and `RUNTIME_UPSTREAM` on the dashboard container's nginx proxy.

## Next steps

- [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack) — bring up admin + runtime + events alongside the dashboard
- [Admin API reference](/admin-api/) — the endpoints the dashboard calls
- [Observability Setup](/how-to/observability-setup) — Prometheus scrape + `cycles_*` metrics
- [Monitoring and Alerting](/how-to/monitoring-and-alerting) — alert rules using the `cycles_*` counters

## Source

- Repository: [`runcycles/cycles-dashboard`](https://github.com/runcycles/cycles-dashboard)
- Image: `ghcr.io/runcycles/cycles-dashboard`
- Stack: Vue 3 + TypeScript + Vite, Pinia, Vue Router, Tailwind CSS v4. Tests: Vitest (unit) + Playwright (E2E against the live compose stack).
- License: Apache 2.0



# Deploying the Events Service

The events service (`cycles-server-events`) has two async jobs: webhook delivery and CyclesEvidence signing. Use webhook delivery to get real-time alerts in Slack, PagerDuty, or your own systems when budgets run out, thresholds are crossed, or reservations are denied. Use CyclesEvidence signing when runtime responses need verifiable audit receipts.

As of v0.1.25.9 the service binds two ports: the application port `7980` and a separate management port `9980` for actuator endpoints (`/actuator/health`, `/actuator/info`, `/actuator/prometheus`). The current reference service is an outbound worker; webhook delivery and evidence signing do not require inbound application traffic. Keep `9980` internal-only for health checks and Prometheus scraping, and do not publish `7980` unless your deployment has an explicit internal control-plane use for that app port.

It is optional — the admin and runtime servers operate normally without it. When deployed, it consumes delivery jobs from Redis and sends HTTP POST requests to webhook endpoints with HMAC-SHA256 signatures. If CyclesEvidence is enabled, it also consumes evidence-source records from Redis, signs envelopes with Ed25519, and stores them by content hash for `GET /v1/evidence/{id}`.

## Quick start with Docker

If you already have the full stack running via [Deploying the Full Cycles Stack](/quickstart/deploying-the-full-cycles-stack), uncomment the `cycles-events` block in your `docker-compose.yml` and restart. Otherwise, use the full-stack compose from the admin repo:

```bash
# From the cycles-server-admin directory
export WEBHOOK_SECRET_ENCRYPTION_KEY=$(openssl rand -base64 32)
docker compose -f docker-compose.full-stack.yml up
```

Services: Redis (6379), Admin (7979), Runtime (7878), Events app port (7980), Events management/actuator (9980).

Note that the admin repo's dev full-stack compose publishes the app port `7980` but not the management port `9980` — the actuator endpoints are only reachable from inside the compose network unless you add a `9980:9980` port mapping.

## Standalone deployment

### From pre-built image

```bash
docker run -d --name cycles-events \
  -e REDIS_HOST=redis.example.com \
  -e REDIS_PORT=6379 \
  -e REDIS_PASSWORD=your-redis-password \
  -e WEBHOOK_SECRET_ENCRYPTION_KEY=your-base64-key \
  ghcr.io/runcycles/cycles-server-events:0.1.25.25
```

The service does not need inbound traffic from applications or webhook targets; it sends webhook HTTP requests outbound. For local inspection, temporarily add `-p 9980:9980` and query the management endpoint from the host. In production, scrape `9980` from Prometheus on an internal network path and leave `7980` unpublished unless you have a specific internal use for the app port.

### From JAR

```bash
REDIS_HOST=redis.example.com \
REDIS_PORT=6379 \
REDIS_PASSWORD=your-redis-password \
WEBHOOK_SECRET_ENCRYPTION_KEY=your-base64-key \
java -jar cycles-server-events-*.jar
```

## Configuration

### Required

| Variable | Description |
|----------|-------------|
| `REDIS_HOST` | Redis hostname (shared with admin and runtime servers) |
| `REDIS_PORT` | Redis port (default: 6379) |
| `REDIS_PASSWORD` | Redis password (empty for no auth) |

### Recommended

| Variable | Default | Description |
|----------|---------|-------------|
| `WEBHOOK_SECRET_ENCRYPTION_KEY` | (required by default) | AES-256-GCM key for signing secret decryption. Base64-encoded 32 bytes. Must match admin and runtime. Generate: `openssl rand -base64 32`. A missing key fails startup unless the local/development-only `WEBHOOK_SECRET_ALLOW_PLAINTEXT=true` escape hatch is set. |

### Optional: CyclesEvidence signing

Configure these only when runtime responses should include verifiable `cycles_evidence` references. The public identity must match the runtime server's `EVIDENCE_SERVER_ID` and `EVIDENCE_SIGNING_SIGNER_DID`; the private key belongs only on `cycles-server-events`.

| Variable | Description |
|----------|-------------|
| `EVIDENCE_SERVER_ID` | Issuer base URL including `/v1`, for example `https://cycles.example.com/v1`. Blank disables evidence signing; pending source records are left untouched, not dead-lettered. |
| `EVIDENCE_SIGNING_SIGNER_DID` | Raw-hex Ed25519 public key. Must match the runtime server's public signer identity. |
| `EVIDENCE_SIGNING_PRIVATE_KEY_HEX` | Raw-hex Ed25519 private key. Keep this secret and deploy it only to `cycles-server-events`. |

The runtime server also publishes public JWKS metadata with `EVIDENCE_SIGNING_KID`, `EVIDENCE_SIGNING_NBF_MS`, and `EVIDENCE_SIGNING_RETIRED_KEYS`. Those variables are not read by the events service.

### Tuning

| Variable | Default | Description |
|----------|---------|-------------|
| `dispatch.pending.timeout-seconds` | 5 | BLMOVE blocking timeout for the delivery queue (seconds; property, not env var) |
| `dispatch.retry.poll-interval-ms` | 5000 | How often to check for ready retries (ms; property, not env var) |
| `dispatch.http.timeout-seconds` | 30 | HTTP request timeout for webhook delivery (property, not env var) |
| `dispatch.http.connect-timeout-seconds` | 5 | HTTP connect timeout (property, not env var) |
| `RETRY_BATCH_SIZE` | 100 | Max due retries claimed per retry poll |
| `DISPATCH_PROCESSING_RECOVERY_IDLE_MS` | 180000 | Idle window before in-flight deliveries on the processing list are recovered back to pending |
| `MANAGEMENT_PORT` | 9980 | Separate management port for the actuator endpoints |
| `MAX_DELIVERY_AGE_MS` | 86400000 | Deliveries older than this auto-fail (24h) |
| `EVENT_TTL_DAYS` | 90 | Redis TTL for event records |
| `DELIVERY_TTL_DAYS` | 14 | Redis TTL for delivery records |
| `RETENTION_CLEANUP_INTERVAL_MS` | 3600000 | ZSET index cleanup interval (1h) |
| `EVIDENCE_POP_TIMEOUT_SECONDS` | 5 | BLMOVE blocking timeout for the evidence queue (seconds) |
| `EVIDENCE_FAILED_MAX_LEN` | 10000 | Max length of the evidence dead-letter queue (newest kept) |

### Full configuration example

Environment variables (Docker `-e` flags or the shell environment):

```bash
REDIS_HOST=redis.example.com
REDIS_PORT=6379
REDIS_PASSWORD=your-redis-password
WEBHOOK_SECRET_ENCRYPTION_KEY=K7x2mP9qR4sT6wB1cD3fG5hJ8kL0nA2=
EVIDENCE_SERVER_ID=https://cycles.example.com/v1
EVIDENCE_SIGNING_SIGNER_DID=b10554...c522
EVIDENCE_SIGNING_PRIVATE_KEY_HEX=4f9c...d20a
MANAGEMENT_PORT=9980
RETRY_BATCH_SIZE=100
DISPATCH_PROCESSING_RECOVERY_IDLE_MS=180000
MAX_DELIVERY_AGE_MS=86400000
EVENT_TTL_DAYS=90
DELIVERY_TTL_DAYS=14
RETENTION_CLEANUP_INTERVAL_MS=3600000
```

Application properties (only overridable via `application.properties` or `-D` system properties — they have no env-var mapping):

```properties
dispatch.pending.timeout-seconds=5
dispatch.retry.poll-interval-ms=5000
dispatch.http.timeout-seconds=30
dispatch.http.connect-timeout-seconds=5
```

## Health check

The events service exposes a Spring Boot Actuator health endpoint on the management port (9980 by default as of v0.1.25.9):

```bash
curl http://localhost:9980/actuator/health
# {"status":"UP"}
```

Pre-v0.1.25.9 deployments exposed `/actuator/health` on the application port 7980. Update kubelet probes and Docker `HEALTHCHECK` commands to hit `:9980` when upgrading. The published Docker image's built-in `HEALTHCHECK` (30s interval, 60s start period, 5 retries) has already been updated.

## What happens when the events service is down

1. **Admin and runtime servers are unaffected** — event emission and evidence source writes are fire-and-forget, never blocking API responses
2. **Events and deliveries accumulate in Redis** — `event:{id}` keys (90-day TTL), `delivery:{id}` keys (14-day TTL), `dispatch:pending` list grows
3. **Redis memory is bounded** — TTLs ensure keys auto-expire even if never consumed
4. **When the events service restarts:**
   - Stale deliveries (older than `MAX_DELIVERY_AGE_MS`, default 24h) are immediately marked FAILED
   - Fresh deliveries are processed normally via the BLMOVE reliable queue — claimed jobs sit on `dispatch:processing` until acknowledged, and orphans idle longer than `DISPATCH_PROCESSING_RECOVERY_IDLE_MS` (default 180000 ms) are recovered back to pending
   - `RetentionCleanupService` trims orphaned ZSET index entries hourly
5. **No data loss for events** — event records persist in Redis for 90 days regardless of delivery status
6. **Evidence may be temporarily unavailable** — responses can still include `cycles_evidence`, but `GET /v1/evidence/{id}` may return transient `404` until the events service signs and stores the envelope

## Auto-disable for persistently failing subscriptions

The events service tracks `consecutive_failures` per subscription. When the counter reaches `disable_after_failures` (default **10**), the subscription transitions to `DISABLED` and no further deliveries are attempted. The counter resets to 0 on any successful delivery. Re-enable a disabled subscription with `PATCH /v1/admin/webhooks/{id}` once the receiver is healthy.

Stale deliveries (older than `MAX_DELIVERY_AGE_MS`, default 24h) are marked `FAILED` without attempting HTTP delivery. This prevents a large backlog from triggering thundering-herd traffic against a receiver after a long events-service outage.

Signing secrets are encrypted at rest with AES-256-GCM using `WEBHOOK_SECRET_ENCRYPTION_KEY` (v0.1.25.2+). The events service decrypts per delivery; plaintext never lives on disk.

## Prometheus metrics

The events service publishes 17 delivery, evidence, dispatcher, and security counters plus one latency timer on `/actuator/prometheus`, served on the management port (9980 by default as of v0.1.25.9; was 7980 on pre-.9 builds).

| Metric | Tags | Description |
|--------|------|-------------|
| `cycles_webhook_delivery_attempts_total` | `tenant`, `event_type` | Every outbound HTTP attempt (including retries) |
| `cycles_webhook_delivery_success_total` | `tenant`, `event_type`, `status_code_family` (`2xx`) | Attempts that received HTTP 2xx |
| `cycles_webhook_delivery_failed_total` | `tenant`, `event_type`, `reason` | Failed attempts, bucketed by failure reason |
| `cycles_webhook_delivery_retried_total` | `tenant`, `event_type` | Retry attempts scheduled on the `dispatch:retry` ZSET |
| `cycles_webhook_delivery_stale_total` | `tenant` | Deliveries auto-failed by the `MAX_DELIVERY_AGE_MS` gate |
| `cycles_webhook_delivery_dead_lettered_total` | `reason` | Delivery jobs moved to dead letter |
| `cycles_webhook_delivery_boundary_skipped_total` | `tenant`, `event_type`, `category` | Deliveries intentionally skipped at a configured boundary |
| `cycles_webhook_subscription_auto_disabled_total` | `tenant`, `reason` | Subscriptions transitioned to `DISABLED` after `disable_after_failures` |
| `cycles_webhook_delivery_latency_seconds` | `tenant`, `event_type`, `outcome` | Timer — HTTP RTT per delivery attempt |
| `cycles_webhook_events_payload_invalid_total` | `type`, `rule` | Event payload validation discrepancies (no tenant tag — shape issue, not traffic) |
| `cycles_evidence_claimed_total` | `artifact_type` | Evidence source records claimed |
| `cycles_evidence_stored_total` | `artifact_type` | Signed evidence envelopes stored |
| `cycles_evidence_dead_lettered_total` | `artifact_type`, `reason` | Evidence source records moved to dead letter |
| `cycles_evidence_retry_deferred_total` | `artifact_type`, `reason` | Evidence work deferred for retry |
| `cycles_webhook_dispatcher_event_published_total` | `event_type` | Dispatcher lifecycle events published |
| `cycles_webhook_dispatcher_event_deferred_total` | `event_type`, `reason` | Dispatcher lifecycle events deferred |
| `cycles_webhook_dispatcher_event_dead_lettered_total` | `event_type`, `reason` | Dispatcher lifecycle events moved to dead letter |
| `cycles_webhook_security_config_indeterminate_total` | — | Delivery blocked because security configuration could not be determined safely |

The `tenant` tag on all counters is gated by `cycles.metrics.tenant-tag.enabled` (default `false` to bound Prometheus cardinality) — set `CYCLES_METRICS_TENANT_TAG_ENABLED=true` to break metrics out per tenant in smaller deployments.

Alert on `cycles_webhook_subscription_auto_disabled_total` (any increase is a receiver health issue) and on a sustained rise in `cycles_webhook_delivery_failed_total{reason!="http_4xx"}` (non-client-error failures indicate dispatch issues). See the [Prometheus Metrics Reference](/how-to/prometheus-metrics-reference) for label values and cardinality guidance.

## Scaling

Multiple events service instances can safely share the same `dispatch:pending` list. `BLMOVE` atomically moves a claimed delivery to `dispatch:processing`, and owner-token-checked acknowledgement prevents a stale worker from removing a successor's claim. Claims idle past `DISPATCH_PROCESSING_RECOVERY_IDLE_MS` (default `180000` ms) are recovered to pending. A fleet-wide ordering lease serializes the claim/send critical section, so additional replicas provide failover rather than higher initial-delivery throughput. Delivery remains at least once; receivers must deduplicate by event ID.

## Next steps

- [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol) — full event type catalog and delivery specification
- [CyclesEvidence Envelopes](/protocol/cycles-evidence-envelopes-in-cycles) — evidence signing, JWKS verification, and rotation
- [Managing Webhooks](/how-to/managing-webhooks) — create, test, and monitor webhooks
- [Webhook Integrations](/how-to/webhook-integrations) — PagerDuty, Slack, ServiceNow examples
- [Configuration Reference](/configuration/server-configuration-reference-for-cycles#events-service-configuration) — all events service settings
- [Architecture Overview](/quickstart/architecture-overview-how-cycles-fits-together) — how the events service fits in the system



# Deploying the Full Cycles Stack

This guide walks you from zero to a working Cycles deployment. By the end, you will have budget enforcement running and verified with a real reserve-commit cycle.

## What you'll have at the end

A working Cycles stack where you can reserve budget, commit actual spend, and verify balances:

```bash
$ curl -s -X POST http://localhost:7878/v1/reservations ...
{ "decision": "ALLOW", "reservation_id": "rsv_..." }

$ curl -s -X POST http://localhost:7878/v1/reservations/rsv_.../commit ...
{ "status": "COMMITTED" }

$ curl -s http://localhost:7878/v1/balances?tenant=acme-corp ...
{ "remaining": ..., "spent": 350000, ... }
```

<details>
<summary><strong>TL;DR — Full quickstart in 60 seconds</strong></summary>

::: warning Production security
The quickstart examples below use empty `REDIS_PASSWORD` and a weak `ADMIN_API_KEY` for fast local development. **For production**, generate strong secrets before deploying:
```bash
export REDIS_PASSWORD=$(openssl rand -base64 32)
export ADMIN_API_KEY=$(openssl rand -base64 32)
export WEBHOOK_SECRET_ENCRYPTION_KEY=$(openssl rand -base64 32)
```
Store these in a secrets manager, not in docker-compose files. Bind the Admin Server to internal network only (`127.0.0.1:7979:7979`). See [Security Hardening](/how-to/security-hardening) for the full checklist.
:::

If you have Docker running and just want to try Cycles immediately, copy-paste this entire block:

```bash
# 1. Create docker-compose.yml and start the stack
cat > docker-compose.yml <<'COMPOSE'
services:
  redis:
    image: redis:7-alpine
    ports: ["6379:6379"]
    volumes: ["redis-data:/data"]
    command: redis-server --appendonly yes
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5
  cycles-admin:
    image: ghcr.io/runcycles/cycles-server-admin:0.1.25.55
    ports: ["7979:7979"]
    environment:
      REDIS_HOST: redis
      REDIS_PORT: 6379
      REDIS_PASSWORD: ""
      ADMIN_API_KEY: admin-bootstrap-key
      WEBHOOK_SECRET_ENCRYPTION_KEY: "${WEBHOOK_SECRET_ENCRYPTION_KEY:?WEBHOOK_SECRET_ENCRYPTION_KEY must be set}"
      WEBHOOK_SECRET_ALLOW_PLAINTEXT: "false"
      DASHBOARD_CORS_ORIGIN: "${DASHBOARD_CORS_ORIGIN:-http://localhost:5173}"
    depends_on:
      redis: { condition: service_healthy }
  cycles-server:
    image: ghcr.io/runcycles/cycles-server:0.1.25.59
    ports: ["7878:7878"]
    environment:
      REDIS_HOST: redis
      REDIS_PORT: 6379
      REDIS_PASSWORD: ""
      # Same value as the admin server's ADMIN_API_KEY. Without it, the
      # protected operational endpoints (aggregate health, Prometheus,
      # API docs) and the admin-on-behalf-of paths return 500
      # "server misconfiguration".
      ADMIN_API_KEY: admin-bootstrap-key
      WEBHOOK_SECRET_ENCRYPTION_KEY: "${WEBHOOK_SECRET_ENCRYPTION_KEY:?WEBHOOK_SECRET_ENCRYPTION_KEY must be set}"
      DASHBOARD_CORS_ORIGIN: "${DASHBOARD_CORS_ORIGIN:-http://localhost:5173}"
    depends_on:
      redis: { condition: service_healthy }
  # Optional: webhook event delivery and evidence signing worker
  cycles-events:
    image: ghcr.io/runcycles/cycles-server-events:0.1.25.25
    environment:
      REDIS_HOST: redis
      REDIS_PORT: 6379
      REDIS_PASSWORD: ""
      WEBHOOK_SECRET_ENCRYPTION_KEY: "${WEBHOOK_SECRET_ENCRYPTION_KEY:?WEBHOOK_SECRET_ENCRYPTION_KEY must be set}"
      WEBHOOK_SECRET_ALLOW_PLAINTEXT: "false"
    depends_on:
      redis: { condition: service_healthy }
volumes:
  redis-data:
COMPOSE

# Generate the shared encryption key required by admin and events
export WEBHOOK_SECRET_ENCRYPTION_KEY=$(openssl rand -base64 32)

docker compose up -d

# 2. Wait for services to be ready (readiness probes are public;
#    aggregate /actuator/health requires X-Admin-API-Key since cycles-server 0.1.25.45)
echo "Waiting for services..."
until curl -sf http://localhost:7878/actuator/health/readiness > /dev/null 2>&1; do sleep 1; done
until curl -sf http://localhost:7979/actuator/health/readiness > /dev/null 2>&1; do sleep 1; done
echo "Services are up."

# 3. Create tenant
curl -s -X POST http://localhost:7979/v1/admin/tenants \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: admin-bootstrap-key" \
  -d '{"tenant_id": "acme-corp", "name": "Acme Corporation"}' | jq .

# 4. Create API key and capture it
API_KEY=$(curl -s -X POST http://localhost:7979/v1/admin/api-keys \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: admin-bootstrap-key" \
  -d '{
    "tenant_id": "acme-corp",
    "name": "quickstart-key",
    "permissions": ["reservations:create","reservations:commit","reservations:release","reservations:extend","reservations:list","balances:read","budgets:write"]
  }' | jq -r '.key_secret')
echo "API Key: $API_KEY"

# 5. Create a budget ($1.00 = 100,000,000 microcents)
curl -s -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $API_KEY" \
  -d '{"scope": "tenant:acme-corp", "unit": "USD_MICROCENTS", "allocated": {"amount": 100000000, "unit": "USD_MICROCENTS"}}' | jq .

# 6. Test: reserve → commit → check balance
RESERVATION_ID=$(curl -s -X POST http://localhost:7878/v1/reservations \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $API_KEY" \
  -d '{
    "idempotency_key": "qs-reserve-001",
    "subject": {"tenant": "acme-corp"},
    "action": {"kind": "llm.completion", "name": "openai:gpt-4o"},
    "estimate": {"amount": 500000, "unit": "USD_MICROCENTS"},
    "ttl_ms": 30000
  }' | jq -r '.reservation_id')
echo "Reserved: $RESERVATION_ID"

curl -s -X POST "http://localhost:7878/v1/reservations/$RESERVATION_ID/commit" \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $API_KEY" \
  -d '{"idempotency_key": "qs-commit-001", "actual": {"amount": 350000, "unit": "USD_MICROCENTS"}}' | jq .

curl -s "http://localhost:7878/v1/balances?tenant=acme-corp" \
  -H "X-Cycles-API-Key: $API_KEY" | jq .

echo ""
echo "Done! Your Cycles stack is running."
echo "  Runtime server: http://localhost:7878"
echo "  Admin server:   http://localhost:7979"
echo "  API key:        $API_KEY"
# Swagger UI: the runtime server's /swagger-ui.html requires the
# X-Admin-API-Key header since 0.1.25.45; the admin server's is disabled
# by default (enable with API_DOCS_ENABLED=true and SWAGGER_ENABLED=true).
```

</details>

## What you are deploying

A complete Cycles deployment has four components that share a single Redis instance:

<ArchDiagram />

| Component | Purpose | Port / access |
|---|---|---|
| **Redis 7+** | Stores all budget state, reservations, and tenant data | 6379 |
| **Cycles Admin Server** | Create tenants, API keys, and budget ledgers. Management plane. | 7979 |
| **Cycles Server** | Runtime budget enforcement. Your app talks to this. | 7878 |
| **Cycles Events Service** | Async webhook delivery with HMAC signing and optional CyclesEvidence signing. | No public inbound; app 7980 and management 9980 stay internal |

Your application only talks to the **Cycles Server** (port 7878). You use the **Admin Server** (port 7979) to set up tenants, keys, and budgets before your app starts enforcing. The **Events Service** is optional and outbound-only for normal operation — it delivers webhook notifications asynchronously and, when evidence is configured, signs CyclesEvidence envelopes for the runtime server to serve. See [Deploying the Events Service](/quickstart/deploying-the-events-service).

::: info Optional: deploy the admin dashboard
For a web UI on top of this stack — operator workflows for tenants, budgets, webhooks, events, audit, and incident response (freeze, suspend, force-release) — also deploy the [Cycles Admin Dashboard](/quickstart/deploying-the-cycles-dashboard). It's a Vue 3 SPA that proxies through to the admin server (and to the runtime server for force-release). Skip if you only need SDK integration.
:::

## Prerequisites

- **Docker** and **Docker Compose** (for the quick path — no Java needed), or
- **Java 21+** and **Maven 3.9+** (for running from source without Docker)
- **Redis 7+** (if not using Docker)

Verify Docker is ready:

```bash
docker --version          # Docker 20+ required
docker compose version    # Docker Compose v2+ required
```

If `docker compose` fails, you may need to install the Docker Compose plugin or use the standalone `docker-compose` binary.

## Step 1: Start the infrastructure

### Option A: Docker Compose from GHCR images (recommended for end users)

The easiest way to deploy is using pre-built images from GitHub Container Registry. No Java or Maven required — Docker pulls the images automatically.

Create a `docker-compose.yml`:

```yaml
services:
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data
    command: redis-server --appendonly yes
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5

  cycles-admin:
    image: ghcr.io/runcycles/cycles-server-admin:0.1.25.55
    ports:
      - "7979:7979"
    environment:
      REDIS_HOST: redis
      REDIS_PORT: 6379
      REDIS_PASSWORD: ""
      ADMIN_API_KEY: ${ADMIN_API_KEY:-admin-bootstrap-key}
      WEBHOOK_SECRET_ENCRYPTION_KEY: ${WEBHOOK_SECRET_ENCRYPTION_KEY:?WEBHOOK_SECRET_ENCRYPTION_KEY must be set}
      WEBHOOK_SECRET_ALLOW_PLAINTEXT: "false"
      DASHBOARD_CORS_ORIGIN: ${DASHBOARD_CORS_ORIGIN:-http://localhost:5173}
    depends_on:
      redis:
        condition: service_healthy

  cycles-server:
    image: ghcr.io/runcycles/cycles-server:0.1.25.59
    ports:
      - "7878:7878"
    environment:
      REDIS_HOST: redis
      REDIS_PORT: 6379
      REDIS_PASSWORD: ""
      # Same value as the admin server's ADMIN_API_KEY. Since 0.1.25.45 the
      # aggregate /actuator/health, Prometheus, and API docs endpoints require
      # this key; leaving it unset makes them return 500 "server
      # misconfiguration" (liveness/readiness probes stay public).
      ADMIN_API_KEY: ${ADMIN_API_KEY:-admin-bootstrap-key}
      WEBHOOK_SECRET_ENCRYPTION_KEY: ${WEBHOOK_SECRET_ENCRYPTION_KEY:?WEBHOOK_SECRET_ENCRYPTION_KEY must be set}
      DASHBOARD_CORS_ORIGIN: ${DASHBOARD_CORS_ORIGIN:-http://localhost:5173}
    depends_on:
      redis:
        condition: service_healthy

  # Optional: webhook delivery and CyclesEvidence signing service. Set
  # the same WEBHOOK_SECRET_ENCRYPTION_KEY used by admin and runtime:
  # export WEBHOOK_SECRET_ENCRYPTION_KEY=$(openssl rand -base64 32)
  # Evidence signing is off unless EVIDENCE_SERVER_ID, EVIDENCE_SIGNING_SIGNER_DID,
  # and the worker-only EVIDENCE_SIGNING_PRIVATE_KEY_HEX are configured.
  # Docs: https://runcycles.io/quickstart/deploying-the-events-service
  # cycles-events:
  #   image: ghcr.io/runcycles/cycles-server-events:0.1.25.25
  #   # No public inbound port is required. Add "9980:9980" only for local
  #   # management inspection; keep it internal in production.
  #   environment:
  #     REDIS_HOST: redis
  #     REDIS_PORT: 6379
  #     REDIS_PASSWORD: ""
  #     WEBHOOK_SECRET_ENCRYPTION_KEY: "${WEBHOOK_SECRET_ENCRYPTION_KEY:?WEBHOOK_SECRET_ENCRYPTION_KEY must be set}"
  #     WEBHOOK_SECRET_ALLOW_PLAINTEXT: "false"
  #   depends_on:
  #     redis:
  #       condition: service_healthy

volumes:
  redis-data:
```

Start the stack:

```bash
docker compose up -d
```

::: tip Version pinning
The examples above pin the current compatible images: admin `0.1.25.55`, server `0.1.25.59`, and events `0.1.25.25`. Check the [current version matrix](/changelog#current-versions) before deploying. Admin, runtime, and events ship on independent release cadences — bumping one does not require bumping the others.
:::

Verify all services are healthy:

```bash
curl -s http://localhost:7878/actuator/health/readiness   # Cycles Server
curl -s http://localhost:7979/actuator/health/readiness   # Admin Server
```

Both should return `{"status":"UP"}`. The readiness probes are public; the aggregate `/actuator/health`, `/actuator/prometheus`, and API docs/Swagger endpoints require the `X-Admin-API-Key` header since cycles-server 0.1.25.45.

### Option B: Docker Compose from source (for development)

The repositories include multi-stage Dockerfiles that build the JARs inside Docker — no local Java or Maven installation required. Each repository includes a `docker-compose.full-stack.yml` that brings up Redis, the Cycles Server, the Admin Server, and the Events Service together.

Clone the repositories side by side (the full-stack compose builds all three from sibling directories):

```bash
git clone https://github.com/runcycles/cycles-server.git
git clone https://github.com/runcycles/cycles-server-admin.git
git clone https://github.com/runcycles/cycles-server-events.git
```

Start the full stack from either repo:

```bash
cd cycles-server-admin
docker compose -f docker-compose.full-stack.yml up -d
```

The multi-stage Docker build compiles the JARs automatically — no manual `mvn package` step needed.

Verify all services are healthy:

```bash
curl -s http://localhost:7878/actuator/health/readiness   # Cycles Server
curl -s http://localhost:7979/actuator/health/readiness   # Admin Server
```

Both should return `{"status":"UP"}`.

### Option C: Running from source

Start Redis:

```bash
docker run -d --name cycles-redis -p 6379:6379 redis:7-alpine
```

Build and start the admin server:

```bash
cd cycles-server-admin/cycles-admin-service
mvn clean package -DskipTests
REDIS_HOST=localhost REDIS_PORT=6379 REDIS_PASSWORD= ADMIN_API_KEY=admin-bootstrap-key \
  java -jar cycles-admin-service-api/target/cycles-admin-service-api-*.jar
```

In a second terminal, build and start the cycles server:

```bash
cd cycles-server/cycles-protocol-service
mvn clean package -DskipTests
REDIS_HOST=localhost REDIS_PORT=6379 \
  java -jar cycles-protocol-service-api/target/cycles-protocol-service-api-*.jar
```

## Step 2: Create a tenant

Every budget and API key belongs to a tenant. Create one using the admin API.

::: info Two authentication headers
The next steps use two different headers. **`X-Admin-API-Key`** is the static bootstrap secret you set in docker-compose (`ADMIN_API_KEY`) — it's used here to create tenants and API keys. **`X-Cycles-API-Key`** is the tenant-scoped key you'll create in Step 3 — it's used for budget operations and runtime calls. For the full mapping of which endpoints use which header, see the [Architecture Overview — Authentication](/quickstart/architecture-overview-how-cycles-fits-together#authentication).
:::

::: tip
This step creates a single tenant for the quickstart. For the full tenant lifecycle — listing, updating, suspending, hierarchical tenants, and more — see [Tenant Creation and Management](/how-to/tenant-creation-and-management-in-cycles).
:::

```bash
curl -s -X POST http://localhost:7979/v1/admin/tenants \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: admin-bootstrap-key" \
  -d '{
    "tenant_id": "acme-corp",
    "name": "Acme Corporation"
  }' | jq .
```

You should see the tenant returned with its details.

## Step 3: Create an API key

Create a tenant-scoped API key. This is the key your application will use in the `X-Cycles-API-Key` header:

```bash
curl -s -X POST http://localhost:7979/v1/admin/api-keys \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: admin-bootstrap-key" \
  -d '{
    "tenant_id": "acme-corp",
    "name": "dev-key",
    "description": "Development key for acme-corp",
    "permissions": [
      "reservations:create",
      "reservations:commit",
      "reservations:release",
      "reservations:extend",
      "reservations:list",
      "balances:read",
      "budgets:write"
    ]
  }' | jq .
```

**Important:** The response includes the full API key (e.g., `cyc_live_...`). Save it — the full secret is only returned once.

```bash
# Save the key for use in later steps
export CYCLES_API_KEY="cyc_live_..."   # paste the key from the response
```

## Step 4: Create a budget

Create a budget ledger for the tenant. Without a budget at any derived scope, reservations fail with `404 NOT_FOUND` ("Budget not found for provided scope"):

```bash
curl -s -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "scope": "tenant:acme-corp",
    "unit": "USD_MICROCENTS",
    "allocated": { "amount": 10000000, "unit": "USD_MICROCENTS" }
  }' | jq .
```

This creates a budget ledger with $0.10 (10,000,000 microcents) available to spend. The `allocated` amount is immediately available as spendable balance.

To add more funds later (e.g., on a schedule or when a customer upgrades), use the fund endpoint:

```bash
curl -s -X POST "http://localhost:7979/v1/admin/budgets/fund?scope=tenant:acme-corp&unit=USD_MICROCENTS" \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "operation": "CREDIT",
    "amount": { "amount": 10000000, "unit": "USD_MICROCENTS" },
    "idempotency_key": "topup-001",
    "reason": "Budget top-up"
  }' | jq .
```

::: info Note
The CREDIT operation adds to the existing balance. If you created the budget with 10M and then credit 10M, the total available becomes 20M.
:::

## Step 5: Verify the full lifecycle

Now test a complete reserve → commit cycle against the **Cycles Server** (port 7878):

```bash
# 1. Reserve
RESERVE_RESPONSE=$(curl -s -X POST http://localhost:7878/v1/reservations \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "idempotency_key": "test-deploy-001",
    "subject": { "tenant": "acme-corp" },
    "action": { "kind": "llm.completion", "name": "openai:gpt-4o" },
    "estimate": { "amount": 500000, "unit": "USD_MICROCENTS" },
    "ttl_ms": 30000,
    "overage_policy": "ALLOW_IF_AVAILABLE"
  }')

echo "$RESERVE_RESPONSE" | jq .
RESERVATION_ID=$(echo "$RESERVE_RESPONSE" | jq -r '.reservation_id')
echo "Reservation ID: $RESERVATION_ID"
```

You should see `"decision": "ALLOW"` and a `reservation_id`.

```bash
# 2. Commit actual spend
curl -s -X POST "http://localhost:7878/v1/reservations/$RESERVATION_ID/commit" \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "idempotency_key": "test-commit-001",
    "actual": { "amount": 350000, "unit": "USD_MICROCENTS" },
    "metrics": {
      "tokens_input": 1200,
      "tokens_output": 800,
      "model_version": "gpt-4o-2024-05"
    }
  }' | jq .
```

You should see `"status": "COMMITTED"`.

```bash
# 3. Check the balance
curl -s "http://localhost:7878/v1/balances?tenant=acme-corp" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" | jq .
```

You should see `spent` has increased and `remaining` has decreased.

**Your deployment is working.** The full reserve-commit-balance cycle completed successfully.

## Optional: Enable CyclesEvidence

The quickstart stack enforces budgets without enabling signed evidence by default. To return `cycles_evidence` refs on runtime responses and make `GET /v1/evidence/{id}` resolve after the async signer runs, configure a shared public identity on `cycles-server` and `cycles-server-events`, and keep the private key only on `cycles-server-events`.

Runtime server:

```yaml
environment:
  EVIDENCE_SERVER_ID: http://localhost:7878/v1
  EVIDENCE_SIGNING_SIGNER_DID: <64-hex-public-ed25519-key>
  EVIDENCE_SIGNING_KID: local-dev-1
  EVIDENCE_SIGNING_NBF_MS: <epoch-ms-rotation-start>
```

Events service:

```yaml
environment:
  EVIDENCE_SERVER_ID: http://localhost:7878/v1
  EVIDENCE_SIGNING_SIGNER_DID: <same-64-hex-public-ed25519-key>
  EVIDENCE_SIGNING_PRIVATE_KEY_HEX: <64-hex-private-ed25519-seed>
```

`EVIDENCE_SIGNING_KID` is a public JWK `kid` label for the runtime server's JWKS endpoint. It is not key material and is not read by `cycles-server-events`. For key generation, coherence checks, and rotation, use the [CyclesEvidence envelope reference](/protocol/cycles-evidence-envelopes-in-cycles) and the [events-service deployment guide](/quickstart/deploying-the-events-service#optional-cyclesevidence-signing).

## Step 6: Connect your application

### Spring Boot (using the Cycles Spring Boot Starter)

Add the dependency:

```xml
<dependency>
    <groupId>io.runcycles</groupId>
    <artifactId>cycles-client-java-spring</artifactId>
    <version>0.3.2</version>
</dependency>
```

Configure your project's `application.yml`:

```yaml
cycles:
  base-url: http://localhost:7878
  api-key: ${CYCLES_API_KEY}
  tenant: acme-corp
```

Annotate methods:

```java
@Service
public class LlmService {

    @Cycles(estimate = "#maxTokens * 10", unit = "USD_MICROCENTS",
            actionKind = "llm.completion", actionName = "openai:gpt-4o")
    public String generate(String prompt, int maxTokens) {
        // Call your LLM provider here
        return callOpenAI(prompt, maxTokens);
    }
}
```

### Python / TypeScript (using the runcycles client)

Install the client:

::: code-group
```bash [Python]
pip install runcycles
```
```bash [TypeScript]
npm install runcycles
```
:::

Use the decorator or higher-order function for automatic reserve/execute/commit:

::: code-group
```python [Python]
from runcycles import CyclesClient, CyclesConfig, cycles, set_default_client

config = CyclesConfig(
    base_url="http://localhost:7878",
    api_key="cyc_live_...",
    tenant="acme-corp",
)
client = CyclesClient(config)
set_default_client(client)

@cycles(estimate=5000, action_kind="llm.completion", action_name="openai:gpt-4o")
def generate(prompt: str) -> str:
    return call_openai(prompt)

result = generate("Hello")
```
```typescript [TypeScript]
import { CyclesClient, CyclesConfig, withCycles, setDefaultClient } from "runcycles";

const config = new CyclesConfig({
  baseUrl: "http://localhost:7878",
  apiKey: "cyc_live_...",
  tenant: "acme-corp",
});
const client = new CyclesClient(config);
setDefaultClient(client);

const generate = withCycles(
  { estimate: 5000, actionKind: "llm.completion", actionName: "openai:gpt-4o" },
  async (prompt: string) => {
    return await callOpenAI(prompt);
  },
);

const result = await generate("Hello");
```
:::

Use `CyclesConfig.from_env()` (Python) or `CyclesConfig.fromEnv()` (TypeScript) to load from `CYCLES_BASE_URL`, `CYCLES_API_KEY`, and `CYCLES_TENANT` environment variables. See the [Python Client quickstart](/quickstart/getting-started-with-the-python-client) or [TypeScript Client quickstart](/quickstart/getting-started-with-the-typescript-client) for full details.

### Any language (raw HTTP)

Any HTTP client can use Cycles. The protocol is language-agnostic:

```python
import requests

CYCLES_URL = "http://localhost:7878"
API_KEY = "cyc_live_..."

# Reserve
resp = requests.post(f"{CYCLES_URL}/v1/reservations", json={
    "idempotency_key": "py-001",
    "subject": {"tenant": "acme-corp"},
    "action": {"kind": "llm.completion", "name": "openai:gpt-4o"},
    "estimate": {"amount": 500000, "unit": "USD_MICROCENTS"},
    "ttl_ms": 30000
}, headers={"X-Cycles-API-Key": API_KEY})

reservation_id = resp.json()["reservation_id"]

# ... call the LLM ...

# Commit
requests.post(f"{CYCLES_URL}/v1/reservations/{reservation_id}/commit", json={
    "idempotency_key": "py-commit-001",
    "actual": {"amount": 420000, "unit": "USD_MICROCENTS"}
}, headers={"X-Cycles-API-Key": API_KEY})
```

## Environment variable reference

### Cycles Server (port 7878)

| Variable | Default | Description |
|---|---|---|
| `REDIS_HOST` | `localhost` | Redis hostname |
| `REDIS_PORT` | `6379` | Redis port |
| `REDIS_PASSWORD` | (empty) | Redis password |
| `server.port` | `7878` | HTTP port |
| `cycles.expiry.interval-ms` | `5000` | Reservation expiry sweep interval (ms) |
| `EVIDENCE_SERVER_ID` | (empty) | Public CyclesEvidence issuer base including `/v1`; set with `EVIDENCE_SIGNING_SIGNER_DID` to emit evidence refs |
| `EVIDENCE_SIGNING_SIGNER_DID` | (empty) | Raw-hex public Ed25519 key; must match `cycles-server-events` when evidence is enabled |
| `EVIDENCE_SIGNING_KID` | derived | Public JWK `kid` label for `GET /v1/.well-known/cycles-jwks.json`; not key material |
| `EVIDENCE_SIGNING_NBF_MS` | `0` | Active JWK validity start, epoch ms |
| `EVIDENCE_SIGNING_RETIRED_KEYS` | (empty) | JSON rotation history for retired public signing keys |

### Cycles Admin Server (port 7979)

| Variable | Default | Description |
|---|---|---|
| `REDIS_HOST` | (required) | Redis hostname |
| `REDIS_PORT` | (required) | Redis port |
| `REDIS_PASSWORD` | (required) | Redis password (set empty string if none) |
| `ADMIN_API_KEY` | (empty) | Master admin key for `X-Admin-API-Key` header |
| `server.port` | `7979` | HTTP port |

### Cycles Events Service (outbound worker)

| Variable | Default | Description |
|---|---|---|
| `REDIS_HOST` | `localhost` | Redis hostname |
| `REDIS_PORT` | `6379` | Redis port |
| `REDIS_PASSWORD` | (empty) | Redis password |
| `MANAGEMENT_PORT` | `9980` | Separate management port for health and Prometheus; keep internal-only |
| `EVENT_TTL_DAYS` | `90` | Event record retention (days) |
| `DELIVERY_TTL_DAYS` | `14` | Webhook delivery record retention (days) |
| `WEBHOOK_SECRET_ENCRYPTION_KEY` | required by default | AES-256-GCM key for webhook signing secrets at rest. Missing key fails admin/events startup unless the local-development-only `WEBHOOK_SECRET_ALLOW_PLAINTEXT=true` escape hatch is set. |
| `EVIDENCE_SERVER_ID` | (empty) | Same issuer base as the runtime server. Blank disables evidence signing and leaves pending evidence records untouched. |
| `EVIDENCE_SIGNING_SIGNER_DID` | (empty) | Same raw-hex public Ed25519 key as the runtime server |
| `EVIDENCE_SIGNING_PRIVATE_KEY_HEX` | (empty) | Raw-hex private Ed25519 key; set only on `cycles-server-events` |

## Troubleshooting

### "NOT_FOUND" (no budget) or "BUDGET_EXCEEDED" on first reservation

`404 NOT_FOUND` means no budget exists at any derived scope — create a budget ledger via the admin API (Step 4); at least one scope in the subject hierarchy needs an allocated budget. `409 BUDGET_EXCEEDED` means a budget exists but the estimate exceeds what remains.

### "UNAUTHORIZED" or 401

The API key is missing, invalid, or expired. Verify with:

```bash
curl -s -X POST http://localhost:7979/v1/auth/validate \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: admin-bootstrap-key" \
  -d '{"key_secret": "cyc_live_..."}' | jq .
```

### Connection refused on port 7878 or 7979

The server is not running. Check Docker containers (`docker compose ps`) or check that the Java processes are running.

### "DEBT_OUTSTANDING" on new reservations

A scope has accumulated debt from `ALLOW_WITH_OVERDRAFT` commits and has no `overdraft_limit` configured (or it is 0). When an `overdraft_limit > 0` is set, debt within the limit does not block reservations. To resolve, either repay the debt or configure an overdraft limit. Repay via the admin API:

```bash
curl -s -X POST "http://localhost:7979/v1/admin/budgets/fund?scope=tenant:acme-corp&unit=USD_MICROCENTS" \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "operation": "REPAY_DEBT",
    "amount": { "amount": 500000, "unit": "USD_MICROCENTS" },
    "idempotency_key": "repay-001"
  }' | jq .
```

### Docker daemon not running

If `docker compose up` fails with "Cannot connect to the Docker daemon", ensure Docker Desktop is running (macOS/Windows) or that the Docker service is started (`sudo systemctl start docker` on Linux).

### Port conflicts

If you see "port is already allocated", another service is using port 6379, 7878, or 7979. Stop the conflicting service or change the port mapping in your `docker-compose.yml` (e.g., `"7879:7878"`).

### Redis connection errors

Ensure Redis 7+ is running and accessible at the configured host:port. Test with:

```bash
redis-cli -h $REDIS_HOST -p $REDIS_PORT ping
```

## Next steps

- [Architecture Overview](/quickstart/architecture-overview-how-cycles-fits-together) — how the components interact in detail
- [Server Configuration Reference](/configuration/server-configuration-reference-for-cycles) — all server configuration properties
- [Getting Started with the MCP Server](/quickstart/getting-started-with-the-mcp-server) — add runtime authority to Claude Desktop, Claude Code, Cursor, or Windsurf
- [Getting Started with the Spring Boot Starter](/quickstart/getting-started-with-the-cycles-spring-boot-starter) — full Spring Boot integration guide
- [Budget Allocation and Management](/how-to/budget-allocation-and-management-in-cycles) — budget patterns and strategies
- [API Key Management](/how-to/api-key-management-in-cycles) — key rotation, scoping, and security



# End-to-End Tutorial: Zero to Budget-Guarded LLM Call

This tutorial takes you from nothing to a working budget-guarded OpenAI call in about 10 minutes. You will deploy the Cycles stack, create a tenant, fund a budget, and make your first budget-enforced LLM call.

::: tip Want to see Cycles in action before building?
Check out the [Demos](/demos/) — self-contained scenarios you can run in 60 seconds, no LLM key required.
:::

## Prerequisites

- **Docker** and **Docker Compose v2+**
- **Python 3.10+** or **Node.js 20+** (for the application step)
- An **OpenAI API key** (for the final step — or use the mock tabs below if you don't have one)

## Quick code preview

Want to see what Cycles integration looks like before setting up the stack? Here is the complete pattern:

::: code-group
```python [Python]
from runcycles import CyclesClient, CyclesConfig, cycles, set_default_client

client = CyclesClient(CyclesConfig(
    base_url="http://localhost:7878",  # Cycles server
    api_key="cyc_live_...",            # from the admin API
    tenant="my-app",
))
set_default_client(client)

@cycles(estimate=2000000, action_kind="llm.completion", action_name="openai:gpt-4o-mini")
def ask(prompt: str) -> str:
    return call_your_llm(prompt)  # any LLM provider

result = ask("Hello")  # Budget reserved → LLM called → cost committed
```
```typescript [TypeScript]
import { CyclesClient, CyclesConfig, withCycles, setDefaultClient } from "runcycles";

const client = new CyclesClient(new CyclesConfig({
  baseUrl: "http://localhost:7878",
  apiKey: "cyc_live_...",
  tenant: "my-app",
}));
setDefaultClient(client);

const ask = withCycles(
  { estimate: 2000000, actionKind: "llm.completion", actionName: "openai:gpt-4o-mini" },
  async (prompt: string) => callYourLlm(prompt),
);

const result = await ask("Hello");  // Budget reserved → LLM called → cost committed
```
:::

::: info
This code requires a running Cycles server. The tutorial below walks you through setting one up with Docker in about 2 minutes. If you just want to see a demo without any setup, check the [Demos](/demos/) page instead.
:::

## Step 1: Start the Cycles stack

Create a `docker-compose.yml` and start the infrastructure:

```bash
cat > docker-compose.yml <<'COMPOSE'
services:
  redis:
    image: redis:7-alpine
    ports: ["6379:6379"]
    command: redis-server --appendonly yes
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5
  cycles-admin:
    image: ghcr.io/runcycles/cycles-server-admin:0.1.25.55
    ports: ["7979:7979"]
    environment:
      REDIS_HOST: redis
      REDIS_PORT: 6379
      REDIS_PASSWORD: ""
      ADMIN_API_KEY: admin-bootstrap-key
      # Local tutorial only. Production must set WEBHOOK_SECRET_ENCRYPTION_KEY.
      WEBHOOK_SECRET_ALLOW_PLAINTEXT: "true"
    depends_on:
      redis: { condition: service_healthy }
  cycles-server:
    image: ghcr.io/runcycles/cycles-server:0.1.25.59
    ports: ["7878:7878"]
    environment:
      REDIS_HOST: redis
      REDIS_PORT: 6379
      REDIS_PASSWORD: ""
      # Same key as the admin server - used for admin-on-behalf-of
      # endpoints and the protected actuator/API-docs surface (unset,
      # those endpoints return 500 server-misconfiguration, not 401)
      ADMIN_API_KEY: admin-bootstrap-key
    depends_on:
      redis: { condition: service_healthy }
COMPOSE

docker compose up -d
```

Wait for services to be healthy (use the readiness probe — since 0.1.25.45 the aggregate `/actuator/health` requires the `X-Admin-API-Key` header, while liveness/readiness stay public):

```bash
until curl -sf http://localhost:7878/actuator/health/readiness > /dev/null 2>&1; do sleep 1; done
until curl -sf http://localhost:7979/actuator/health/readiness > /dev/null 2>&1; do sleep 1; done
echo "Cycles is running."
```

## Step 2: Create a tenant

::: info Two authentication headers
Steps 2-3 use `X-Admin-API-Key` — the static bootstrap secret set in docker-compose (`ADMIN_API_KEY`). Steps 4-5 switch to `X-Cycles-API-Key` — the tenant-scoped key created in Step 3. See [Authentication](/quickstart/architecture-overview-how-cycles-fits-together#authentication) for why.
:::

```bash
curl -s -X POST http://localhost:7979/v1/admin/tenants \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: admin-bootstrap-key" \
  -d '{"tenant_id": "my-app", "name": "My Application"}' | jq .
```

## Step 3: Create an API key

```bash
API_KEY=$(curl -s -X POST http://localhost:7979/v1/admin/api-keys \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: admin-bootstrap-key" \
  -d '{
    "tenant_id": "my-app",
    "name": "tutorial-key",
    "permissions": ["reservations:create","reservations:commit","reservations:release","reservations:extend","reservations:list","balances:read","budgets:write"]
  }' | jq -r '.key_secret')

echo "Your API key: $API_KEY"
```

Save this key — the secret is only shown once.

## Step 4: Create a budget

Give the tenant $1.00 (100,000,000 microcents) to spend:

```bash
curl -s -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $API_KEY" \
  -d '{
    "scope": "tenant:my-app",
    "unit": "USD_MICROCENTS",
    "allocated": { "amount": 100000000, "unit": "USD_MICROCENTS" }
  }' | jq .
```

## Step 5: Verify with a raw HTTP test

Before adding an SDK, confirm the lifecycle works with curl:

```bash
# Reserve
RESERVATION_ID=$(curl -s -X POST http://localhost:7878/v1/reservations \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $API_KEY" \
  -d '{
    "idempotency_key": "tutorial-001",
    "subject": {"tenant": "my-app"},
    "action": {"kind": "llm.completion", "name": "test"},
    "estimate": {"amount": 500000, "unit": "USD_MICROCENTS"},
    "ttl_ms": 30000
  }' | jq -r '.reservation_id')
echo "Reserved: $RESERVATION_ID"

# Commit
curl -s -X POST "http://localhost:7878/v1/reservations/$RESERVATION_ID/commit" \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $API_KEY" \
  -d '{"idempotency_key": "tutorial-commit-001", "actual": {"amount": 350000, "unit": "USD_MICROCENTS"}}' | jq .

# Check balance
curl -s "http://localhost:7878/v1/balances?tenant=my-app" \
  -H "X-Cycles-API-Key: $API_KEY" | jq .
```

You should see `"decision": "ALLOW"`, then `"status": "COMMITTED"`, then a balance with `spent` of 350,000 and the remaining budget reduced accordingly.

## Step 6: Build a budget-guarded application

Choose your language:

::: code-group
```python [Python]
# Install: pip install runcycles openai
# Save as app.py

import os
from openai import OpenAI
from runcycles import CyclesClient, CyclesConfig, cycles, set_default_client

# Configure Cycles
cycles_client = CyclesClient(CyclesConfig(
    base_url="http://localhost:7878",
    api_key=os.environ["CYCLES_API_KEY"],
    tenant="my-app",
))
set_default_client(cycles_client)

# Configure OpenAI
openai_client = OpenAI()

@cycles(
    estimate=2000000,  # Reserve $0.02 per call
    action_kind="llm.completion",
    action_name="openai:gpt-4o-mini",
)
def ask(prompt: str) -> str:
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=500,
    )
    return response.choices[0].message.content

# Run it
try:
    result = ask("What is budget governance for AI agents? Reply in one sentence.")
    print(f"Response: {result}")
except Exception as e:
    print(f"Error: {e}")
```
```typescript [TypeScript]
// Install: npm init -y && npm install runcycles openai
// Save as app.ts

import OpenAI from "openai";
import { CyclesClient, CyclesConfig, withCycles, setDefaultClient } from "runcycles";

const cyclesClient = new CyclesClient(new CyclesConfig({
  baseUrl: "http://localhost:7878",
  apiKey: process.env.CYCLES_API_KEY!,
  tenant: "my-app",
}));
setDefaultClient(cyclesClient);

const openai = new OpenAI();

const ask = withCycles(
  {
    estimate: 2000000,
    actionKind: "llm.completion",
    actionName: "openai:gpt-4o-mini",
  },
  async (prompt: string) => {
    const response = await openai.chat.completions.create({
      model: "gpt-4o-mini",
      messages: [{ role: "user", content: prompt }],
      max_tokens: 500,
    });
    return response.choices[0].message.content;
  },
);

const result = await ask("What is budget governance for AI agents? Reply in one sentence.");
console.log("Response:", result);
```
```python [Python (mock)]
# Install: pip install runcycles
# Save as app_mock.py — no OpenAI key needed

import os
from runcycles import CyclesClient, CyclesConfig, cycles, set_default_client

# Configure Cycles
cycles_client = CyclesClient(CyclesConfig(
    base_url="http://localhost:7878",
    api_key=os.environ["CYCLES_API_KEY"],
    tenant="my-app",
))
set_default_client(cycles_client)

@cycles(
    estimate=2000000,  # Reserve $0.02 per call
    action_kind="llm.completion",
    action_name="mock:gpt-4o-mini",
)
def ask(prompt: str) -> str:
    # Simulated LLM response — no API key required
    return f"[Mock response to: {prompt[:50]}]"

# Run it
try:
    result = ask("What is budget governance for AI agents? Reply in one sentence.")
    print(f"Response: {result}")
except Exception as e:
    print(f"Error: {e}")
```
```typescript [TypeScript (mock)]
// Install: npm init -y && npm install runcycles
// Save as app_mock.ts — no OpenAI key needed

import { CyclesClient, CyclesConfig, withCycles, setDefaultClient } from "runcycles";

const cyclesClient = new CyclesClient(new CyclesConfig({
  baseUrl: "http://localhost:7878",
  apiKey: process.env.CYCLES_API_KEY!,
  tenant: "my-app",
}));
setDefaultClient(cyclesClient);

const ask = withCycles(
  {
    estimate: 2000000,
    actionKind: "llm.completion",
    actionName: "mock:gpt-4o-mini",
  },
  async (prompt: string) => {
    // Simulated LLM response — no API key required
    return `[Mock response to: ${prompt.slice(0, 50)}]`;
  },
);

const result = await ask("What is budget governance for AI agents? Reply in one sentence.");
console.log("Response:", result);
```
:::

::: tip No OpenAI key?
The **mock tabs** replace the OpenAI call with a stub that returns a fixed string. The Cycles budget lifecycle (reserve → commit → balance deduction) works exactly the same — you just skip the LLM cost.
:::

Run it:

```bash
export CYCLES_API_KEY="cyc_live_..."   # your key from Step 3

# With OpenAI:
export OPENAI_API_KEY="sk-..."
python app.py                           # or: npx tsx app.ts

# Without OpenAI (mock):
python app_mock.py                      # or: npx tsx app_mock.ts
```

## Step 7: Watch the budget decrease

After running your app, check the balance again:

```bash
curl -s "http://localhost:7878/v1/balances?tenant=my-app" \
  -H "X-Cycles-API-Key: $API_KEY" | jq '.balances[] | {scope, remaining, spent, reserved}'
```

You'll see `spent` has increased by the actual usage from your LLM call, and `remaining` has decreased.

## Step 8: See what happens when budget runs out

Try exhausting the budget to see enforcement in action. Set a tiny budget on a new scope:

```bash
curl -s -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $API_KEY" \
  -d '{
    "scope": "tenant:my-app/workspace:demo",
    "unit": "USD_MICROCENTS",
    "allocated": { "amount": 100, "unit": "USD_MICROCENTS" }
  }' | jq .
```

Now try to reserve more than the budget:

```bash
curl -s -X POST http://localhost:7878/v1/reservations \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $API_KEY" \
  -d '{
    "idempotency_key": "exceed-001",
    "subject": {"tenant": "my-app", "workspace": "demo"},
    "action": {"kind": "llm.completion", "name": "test"},
    "estimate": {"amount": 500000, "unit": "USD_MICROCENTS"},
    "ttl_ms": 30000
  }' | jq .
```

You'll see `"error": "BUDGET_EXCEEDED"` — the call was blocked *before* any money was spent.

## Cleanup

```bash
docker compose down -v
```

## Common issues

- **`Cannot connect to the Docker daemon`** — Docker Desktop isn't running. Start it and re-run `docker compose up -d`.
- **`bind: address already in use` on port 7878 or 7979** — another service is using these ports. Either stop it, or remap the ports in `docker-compose.yml` (e.g. `"17878:7878"`) and update the curl URLs accordingly.
- **`401 Unauthorized` on Step 2 or 3** — you're using `X-Cycles-API-Key` instead of `X-Admin-API-Key`. Bootstrap calls (creating tenants and API keys) require the admin header. Tenant calls (Steps 4–8) use the Cycles header.
- **`jq: command not found`** — install jq (`brew install jq` on macOS, `apt install jq` on Debian/Ubuntu, `winget install jqlang.jq` on Windows). Or pipe to `python -m json.tool` instead.
- **`tenant_id mismatch` or `tenant not found`** — every step uses `my-app` as the tenant ID. If you changed it in Step 2, update it everywhere else too (including the `subject.tenant` field in reservations).
- **Healthcheck loop never returns** — the `until curl -sf ...` loop is waiting for `/actuator/health/readiness` to return 200. Check `docker compose logs cycles-server` for startup errors (most often a Redis connection issue). If you queried the aggregate `/actuator/health` instead and got `401`, that is expected — it requires `X-Admin-API-Key` since 0.1.25.45.

## Next steps

::: tip Want real-time budget alerts?
The tutorial above deploys the core budget enforcement stack. To receive webhook notifications when budgets run out, thresholds are crossed, or reservations are denied, add the optional events service. It takes 2 minutes.
:::

- [Deploy the Events Service](/quickstart/deploying-the-events-service) — get Slack, PagerDuty, or custom webhook alerts for budget events
- [Python Client Quickstart](/quickstart/getting-started-with-the-python-client) — `@cycles` decorator deep dive
- [TypeScript Client Quickstart](/quickstart/getting-started-with-the-typescript-client) — `withCycles` wrapper deep dive
- [Spring Boot Quickstart](/quickstart/getting-started-with-the-cycles-spring-boot-starter) — `@Cycles` annotation deep dive
- [Demos](/demos/) — see Cycles in action with the runaway agent and action authority scenarios
- [Choose a First Rollout](/quickstart/how-to-choose-a-first-cycles-rollout-tenant-budgets-run-budgets-or-model-call-guardrails) — decide your adoption strategy
- [Adding Cycles to an Existing Application](/how-to/adding-cycles-to-an-existing-application) — integrate incrementally
- [Cost Estimation Cheat Sheet](/how-to/cost-estimation-cheat-sheet) — how much to reserve per model call



# Getting Started with the Cycles Spring Boot Starter

[![Maven Central](https://img.shields.io/maven-central/v/io.runcycles/cycles-client-java-spring?label=Maven%20Central&color=555&style=flat-square)](https://central.sonatype.com/artifact/io.runcycles/cycles-client-java-spring)

The Cycles Spring Boot Starter provides a declarative way to add budget enforcement to any Spring application.

Instead of manually calling the Cycles API for every reservation, commit, and release, the starter provides an `@Cycles` annotation that handles the full lifecycle automatically.

::: tip Using Python?
See the [Python Client quickstart](/quickstart/getting-started-with-the-python-client) instead.
:::

## What the starter does

The starter wraps any annotated method in a reserve → execute → commit lifecycle:

1. **Before the method runs:** evaluates the estimate, creates a reservation, and checks the decision
2. **While the method runs:** maintains the reservation with automatic heartbeat extensions
3. **After the method returns:** commits actual usage and releases any unused remainder
4. **If the method throws:** releases the reservation to return budget to the pool

Once actual usage is known, the starter persists settlement before the first commit request. Ambiguous outcomes replay with the same key, and an expired commit is recovered through `POST /v1/events`. The guarantee cannot cover a JVM exit before actual usage is known; see [SDK Settlement Recovery and Durability](/protocol/sdk-settlement-recovery-and-durability).

::: tip Cycles provides three runtime-authority pillars
- **Spend** — reserve-commit budget enforcement before instrumented LLM calls and tool actions
- **Risky actions** — callers can budget assigned `RISK_POINTS`; applications must apply preflight decisions and any configured caps
- **Audit** — reservations, commits, releases, and direct-usage events create lifecycle records; non-persisting preflight decisions need application logging
:::

All of this happens transparently through Spring AOP.

## Try the demo app first

The fastest way to see the starter in action is to run the included demo application.

### Prerequisites

You need a running Cycles stack with a tenant, API key, and budget. If you don't have one yet, follow [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack) first. The demo app expects the same `acme-corp` tenant used in that guide.

::: tip Where do I get my API key?
API keys are created through the **Cycles Admin Server** (port 7979) and always start with `cyc_live_`. If your stack is already running with a tenant, create one directly:

```bash
curl -s -X POST http://localhost:7979/v1/admin/api-keys \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: admin-bootstrap-key" \
  -d '{
    "tenant_id": "acme-corp",
    "name": "dev-key",
    "permissions": ["reservations:create","reservations:commit","reservations:release","reservations:extend","reservations:list","balances:read"]
  }' | jq -r '.key_secret'
```

The response returns the full key (e.g. `cyc_live_abc123...`). **Save it — the secret is only shown once.**

Need the full setup? See [Deploy the Full Stack — Create an API key](/quickstart/deploying-the-full-cycles-stack#step-3-create-an-api-key). For rotation and lifecycle details, see [API Key Management](/how-to/api-key-management-in-cycles).
:::

### Run the demo

```bash
git clone https://github.com/runcycles/cycles-spring-boot-starter.git
cd cycles-spring-boot-starter/cycles-demo-client-java-spring
```

Set your API key — the `cyc_live_...` value returned when you created the key via the Admin Server:

```bash
export CYCLES_API_KEY=cyc_live_...   # from Admin Server /v1/admin/api-keys response
mvn spring-boot:run
```

Or edit `application.yml` directly and paste the key in `cycles.api-key`.

The demo app starts on port 7955. Try the simplest example first:

```bash
curl -X POST http://localhost:7955/api/demo/annotation/minimal?input=hello
```

Hit `GET http://localhost:7955/api/demo/index` for a full listing of all endpoints with copy-paste curl commands.

### What the demo covers

The demo app includes working examples for every major feature area:

**Start here (`/api/demo/annotation/minimal`)**
- `@Cycles("1000")` — the simplest possible usage, fixed estimate with all defaults

**Annotation-based (`/api/llm/*`)**
- `@Cycles` with SpEL estimate/actual, `CyclesContextHolder` for reading reservation context, `CyclesMetrics` for reporting token counts and latency, and `commitMetadata` for audit data

**Annotation variations (`/api/demo/annotation/*`)**
- `ALLOW_WITH_CAPS` — reading and respecting operator-configured constraints returned by the server — `POST /api/demo/annotation/caps`
- `unit=TOKENS` with `actionTags` — `POST /api/demo/annotation/tokens`
- `unit=CREDITS` with `workflow`, `agent`, and custom `dimensions` — `POST /api/demo/annotation/credits`
- Per-annotation budget scope targeting (`workspace`/`app` override) — `POST /api/demo/annotation/budget-targeting`
- `overagePolicy=ALLOW_WITH_OVERDRAFT` — `POST /api/demo/annotation/overdraft`
- Custom `ttlMs` and `gracePeriodMs` — `POST /api/demo/annotation/custom-ttl`
- `dryRun=true` (shadow-mode evaluation) — `POST /api/demo/annotation/dry-run`

**Programmatic CyclesClient (`/api/demo/client/*`)**
- Full reserve → commit lifecycle — `POST /api/demo/client/reserve-commit`
- Reserve → release (cancellation) — `POST /api/demo/client/reserve-release`
- Preflight decision check — `POST /api/demo/client/decide`
- Balance queries — `GET /api/demo/client/balances`
- Reservation listing — `GET /api/demo/client/reservations`

**Standalone events (`/api/demo/events/*`)**
- Direct debit without reservation — `POST /api/demo/events/record`

**Error handling**
- Global `@RestControllerAdvice` for `CyclesProtocolException` with structured JSON error responses

::: info Note
The deployment guide creates a `USD_MICROCENTS` budget. The `unit=TOKENS` and `unit=CREDITS` demo endpoints require separate budget ledgers for those units. If you only created the default budget, those endpoints will return `BUDGET_EXCEEDED`. Start with the `USD_MICROCENTS` endpoints (minimal, caps, overdraft, custom-ttl, dry-run) and create additional budgets via the admin API if you want to explore other units.
:::

### Suggested walkthrough

Follow this order to build understanding progressively. Each step introduces one new concept.

**1. Reserve and commit with a fixed estimate**

```bash
curl -X POST http://localhost:7955/api/demo/annotation/minimal?input=hello
```

This is `@Cycles("1000")` — the simplest annotation. The response shows the reservation lifecycle result.

**2. Check your balance**

```bash
curl http://localhost:7955/api/demo/client/balances
```

You should see `spent` increased by 1000 microcents from step 1.

**3. Try a dry run (no budget consumed)**

```bash
curl -X POST http://localhost:7955/api/demo/annotation/dry-run?amount=500
```

The server evaluates the reservation but doesn't persist it. Check balances again — they haven't changed.

**4. See how overdraft works**

```bash
curl -X POST http://localhost:7955/api/demo/annotation/overdraft?amount=1000
```

With `overagePolicy=ALLOW_WITH_OVERDRAFT`, the reservation succeeds even if it exceeds the budget.

**5. Use the programmatic client**

```bash
curl -X POST http://localhost:7955/api/demo/client/reserve-commit?estimate=5000
```

This does the same reserve → commit lifecycle as `@Cycles`, but using `CyclesClient` directly. Compare with step 1 to see the annotation vs programmatic approach.

**6. Cancel a reservation**

```bash
curl -X POST http://localhost:7955/api/demo/client/reserve-release?estimate=3000
```

The reservation is created then released — budget is returned to the pool. Check balances to confirm it was refunded.

**7. Preflight check without reserving**

```bash
curl -X POST http://localhost:7955/api/demo/client/decide?estimate=10000
```

The `decide` endpoint tells you whether a reservation *would* be allowed, without creating one.

**8. Record a standalone event**

```bash
curl -X POST 'http://localhost:7955/api/demo/events/record?amount=1500&description=API+call'
```

Direct debit — no reservation needed. Useful for post-hoc usage recording.

After this walkthrough, explore the remaining endpoints (`caps`, `custom-ttl`, `llm/generate`) and read the source files below to see how each feature is implemented.

### Demo app source files

| File | What it demonstrates |
|---|---|
| `service/LlmService.java` | `@Cycles` annotation, `CyclesContextHolder`, `CyclesMetrics`, `commitMetadata` |
| `service/AnnotationShowcaseService.java` | Annotation variations: minimal, caps-aware, units, TTL, overdraft, dry-run, dimensions, budget scope targeting |
| `service/ProgrammaticClientService.java` | Direct `CyclesClient` usage for the full reservation lifecycle |
| `service/EventService.java` | Standalone events via `CyclesClient.createEvent()` |
| `error/CyclesExceptionHandler.java` | Global error handling for `CyclesProtocolException` |
| `resolvers/CyclesTenantResolver.java` | Dynamic tenant resolution via `CyclesFieldResolver` |
| `controller/DemoController.java` | REST endpoints wiring all services at `/api/demo/*` |
| `controller/LlmController.java` | LLM endpoints with budget error handling |

All demo source files are under `cycles-demo-client-java-spring/src/main/java/io/runcycles/demo/client/spring/`.

## Installation

Add the starter to your project:

::: code-group
```xml [Maven]
<dependency>
    <groupId>io.runcycles</groupId>
    <artifactId>cycles-client-java-spring</artifactId>
    <version>0.3.3</version>
</dependency>
```
```groovy [Gradle]
implementation 'io.runcycles:cycles-client-java-spring:0.3.3'
```
:::

Check [Maven Central](https://central.sonatype.com/artifact/io.runcycles/cycles-client-java-spring) for the latest version.

## Configuration

Configure the connection in your project's `application.yml`:

```yaml
cycles:
  base-url: http://localhost:7878
  api-key: your-api-key
  tenant: acme
  workspace: production
  app: support-bot
```

These defaults apply to all `@Cycles`-annotated methods unless overridden per method.

### Optional configuration

```yaml
cycles:
  http:
    connect-timeout: 2s
    read-timeout: 5s
  retry:
    enabled: true
    max-attempts: 5
    initial-delay: 500ms
    multiplier: 2.0
    max-delay: 30s
```

## The @Cycles annotation

The `@Cycles` annotation is applied to methods:

```java
@Cycles("500")
public String summarize(String text) {
    return chatModel.call(text);
}
```

This reserves 500 units (default unit: USD_MICROCENTS) before `summarize()` runs, then commits actual usage afterward.

### SpEL expressions for dynamic estimates

The estimate can use Spring Expression Language to compute cost from method arguments:

```java
@Cycles("#tokens * 10")
public String generate(int tokens) {
    return chatModel.call(prompt, tokens);
}
```

The expression is evaluated before the method runs, using method parameters as variables.

::: warning Compiler flag required for named parameters
SpEL expressions that reference parameters by name (e.g., `#tokens`) require the `-parameters` javac compiler flag. Without it, you'll get a SpEL evaluation error at runtime. Add this to your `pom.xml`:

```xml
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-compiler-plugin</artifactId>
  <configuration>
    <parameters>true</parameters>
  </configuration>
</plugin>
```

Alternatively, use positional references: `#p0`, `#p1`, etc. See the [SpEL Expression Reference](/configuration/spel-expression-reference-for-cycles) for details.
:::

### Specifying actual usage

By default, the estimate is used as the actual amount at commit time. To calculate actual usage from the return value:

```java
@Cycles(estimate = "5000", actual = "#result.usage.totalTokens * 8")
public ChatResponse chat(String prompt) {
    return chatModel.call(prompt);
}
```

The `actual` expression is evaluated after the method returns, with `#result` bound to the return value.

In v0.3.3+, an invalid explicit `actual` expression cannot turn completed work
into a release: the starter logs the evaluation failure, commits the estimate,
and adds `metadata.actual_source=estimate`. If estimate fallback is disabled
and no `actual` expression is configured, validation fails before the
reservation or method execution.

## Annotation attributes

### Subject fields

Override the defaults from configuration:

```java
@Cycles(value = "1000",
         tenant = "acme",
         workspace = "production",
         app = "support-bot",
         workflow = "refund-assistant",
         agent = "planner",
         toolset = "search-tools")
```

Subject fields determine which budget scope the reservation targets. In most cases, set `tenant`, `workspace`, and `app` in configuration and only override specific fields per-method:

```java
// Given config: tenant=acme, workspace=production, app=support-bot

// Uses config defaults → scope: tenant:acme/workspace:production/app:support-bot
@Cycles("500")
public String handleTicket(String text) { ... }

// Overrides workspace → scope: tenant:acme/workspace:staging/app:support-bot
@Cycles(value = "500", workspace = "staging")
public String handleTicketStaging(String text) { ... }
```

The second method targets the **staging** budget scope instead of production. All other fields (`tenant`, `app`) still come from config. Budget scopes are independent — each has its own allocated budget.

Since 0.2.1, subject fields also accept SpEL: a value whose first non-whitespace character is `#` is evaluated against the method invocation, so `tenant = "#tenantId"` resolves the tenant from a method argument. Literal values are passed through unchanged.

### Action identity

```java
@Cycles(value = "1000",
         actionKind = "llm.completion",
         actionName = "openai:gpt-4o-mini",
         actionTags = {"prod", "customer-facing"})
```

If not specified, `actionKind` defaults to the declaring class name and `actionName` defaults to the method name.

### Unit

```java
@Cycles(value = "2500", unit = "TOKENS")
```

Supported units: `USD_MICROCENTS` (default), `TOKENS`, `CREDITS`, `RISK_POINTS`.

### TTL and grace period

```java
@Cycles(value = "1000", ttlMs = 30000, gracePeriodMs = 10000)
```

Default TTL is 60 seconds. When the server returns `remaining_ttl_ms`, the starter schedules from that authoritative lead and reserves enough time for one failed extend, a same-key retry, and margin. Older servers use a best-effort fallback. See [TTL, Grace Period, and Extend](/protocol/reservation-ttl-grace-period-and-extend-in-cycles).

### Overage policy

```java
@Cycles(value = "1000", overagePolicy = "ALLOW_IF_AVAILABLE")
```

Options: `ALLOW_IF_AVAILABLE` (default), `REJECT`, `ALLOW_WITH_OVERDRAFT`.

### Dry run (shadow mode)

```java
@Cycles(value = "1000", dryRun = true)
```

Evaluates the reservation without actually holding budget. The guarded method does **not** execute — instead, the call returns a `DryRunResult` with the full evaluation data: decision, caps, affected scopes, scope path, reserved amount, balances, reason code, and retry hint. A dry-run `DENY` throws `CyclesProtocolException`, just like a real denial. Because the return value is a `DryRunResult` rather than the method's normal result, declare dry-run methods with an `Object` return type. Useful for shadow-mode rollouts where you want to measure budget impact without affecting production behavior.

### Custom dimensions

```java
@Cycles(value = "1000", dimensions = {"cost_center=engineering", "run=run-12345"})
```

### Commit metadata

```java
@Cycles(value = "1000", metadata = "{'app_request_id': #requestId, 'model': #result.model}")
```

Since 0.2.5. The `metadata` SpEL expression is evaluated after the method returns — `#result` is available — and must yield a `Map<String, Object>`. The result is merged with metadata set programmatically via `CyclesContextHolder`; programmatic metadata wins on key conflicts.

## Accessing reservation context at runtime

Inside an annotated method, the current reservation context is available via `CyclesContextHolder`:

```java
@Cycles("1000")
public String process(String input) {
    CyclesReservationContext ctx = CyclesContextHolder.get();

    // Check reservation details
    String reservationId = ctx.getReservationId();
    Decision decision = ctx.getDecision();

    // Check caps (if ALLOW_WITH_CAPS)
    if (ctx.hasCaps()) {
        Caps caps = ctx.getCaps();
        Integer maxTokens = caps.getMaxTokens();
        if (!caps.isToolAllowed("web.search")) {
            // skip web search
        }
    }

    // Check expiration
    if (ctx.isExpiringSoon(5000)) {
        // wrap up quickly
    }

    // Attach metrics for the commit
    CyclesMetrics metrics = new CyclesMetrics();
    metrics.setTokensInput(150);
    metrics.setTokensOutput(80);
    metrics.setLatencyMs(320);
    metrics.setModelVersion("gpt-4o-mini-2024-07-18");
    ctx.setMetrics(metrics);

    // Attach metadata for audit
    ctx.setCommitMetadata(Map.of("app_request_id", "req-abc-123"));

    return chatModel.call(input);
}
```

## Decision handling

When the reservation decision comes back, the starter handles each case:

### ALLOW

The method runs normally.

### ALLOW_WITH_CAPS

The method runs, and a warning is logged. Caps are available through `CyclesContextHolder` for the method to inspect and respect.

### DENY

The method does not run. A `CyclesProtocolException` is thrown with the reason code and optional `retryAfterMs` hint.

The caller can catch this to implement degradation:

```java
try {
    return service.summarize(text);
} catch (CyclesProtocolException e) {
    if (e.getRetryAfterMs() != null) {
        // retry after suggested delay
    }
    return fallbackResponse();
}
```

## Self-invocation (internal method calls)

Spring's proxy-based AOP **does not intercept internal method calls** within the same class. If a method calls another method in the same bean using `this.method()`, the call bypasses the proxy and the `@Cycles` aspect never fires.

```java
// BROKEN — @Cycles is silently ignored on internal calls
@Service
public class MyService {

    public String handleRequest(String input) {
        return guardedCall(input);  // calls this.guardedCall() — bypasses proxy
    }

    @Cycles("#input.length() * 10")
    public String guardedCall(String input) {
        return "Processed: " + input;  // @Cycles never activates
    }
}
```

### Workaround 1: Extract to a separate bean (recommended)

Move the `@Cycles`-annotated method into its own `@Service` and inject it:

```java
@Service
public class GuardedService {

    @Cycles("#input.length() * 10")
    public String guardedCall(String input) {
        return "Processed: " + input;  // @Cycles works — called through proxy
    }
}

@Service
public class MyService {

    @Autowired
    private GuardedService guardedService;

    public String handleRequest(String input) {
        return guardedService.guardedCall(input);
    }
}
```

### Workaround 2: Self-inject the proxy

If extracting a bean is impractical, inject the proxy of your own class using `@Lazy`:

```java
@Service
public class MyService {

    @Lazy
    @Autowired
    private MyService self;

    public String handleRequest(String input) {
        return self.guardedCall(input);  // calls through proxy — @Cycles works
    }

    @Cycles("#input.length() * 10")
    public String guardedCall(String input) {
        return "Processed: " + input;
    }
}
```

::: tip Startup warning
The starter logs a `WARN` at startup when it detects a bean where some methods have `@Cycles` and others do not, since this pattern is susceptible to self-invocation issues. The warning is informational — it does not block startup.
:::

## Nesting prevention

Calling a `@Cycles`-annotated method from inside another `@Cycles`-annotated method — even across different beans — throws an `IllegalStateException`. This is intentional:

- **Double-counting:** The outer reservation already reserves budget for the full operation. An inner reservation would deduct additional budget from the same pool.
- **Protocol design:** The Cycles Protocol v0 has no concept of parent/child reservations. Each reservation is independent and atomic.

```java
// BROKEN — throws IllegalStateException("Nested @Cycles not supported")
@Service
public class Orchestrator {

    @Autowired private LlmService llmService;

    @Cycles("#tokens * 10")
    public String orchestrate(int tokens) {
        return llmService.generate("hello", tokens);  // throws!
    }
}

@Service
public class LlmService {

    @Cycles("#tokens * 5")  // ← second @Cycles while outer is active
    public String generate(String prompt, int tokens) { ... }
}
```

**Correct pattern:** Place `@Cycles` at the outermost entry point only. Inner services should be plain methods:

```java
@Service
public class Orchestrator {

    @Autowired private LlmService llmService;

    @Cycles("#tokens * 10")
    public String orchestrate(int tokens) {
        return llmService.generate("hello", tokens);  // works
    }
}

@Service
public class LlmService {

    // No @Cycles here — called from within an already-guarded operation
    public String generate(String prompt, int tokens) { ... }
}
```

## Commit retry

Known actual usage is written to the durable journal before the first commit request. Transient and ambiguous outcomes retry with the original idempotency key; retry exhaustion, authentication failure, and unclassifiable 4xx responses remain queued across JVM restart. If the reservation expired, the starter switches the durable record to a direct event before attempting recovery. A recognized terminal commit rejection stops retry and discards the unrecoverable journal entry, but never releases the reservation after the guarded method has spent the resource.

The retry engine is configurable and extensible. The default `JournaledCommitRetryEngine` uses exponential backoff and drains for a bounded time during Spring shutdown.

Custom retry strategies can be provided by implementing the `CommitRetryEngine` interface.

## Lifecycle summary

For each `@Cycles`-annotated method call:

1. Estimate is evaluated (SpEL expression or fixed value)
2. Reservation is created on the Cycles server
3. Decision is checked (ALLOW / ALLOW_WITH_CAPS / DENY)
4. If DENY: throw exception, method does not run
5. Heartbeat extension is scheduled (background thread)
6. Method executes
7. Actual usage is evaluated (SpEL expression or estimate)
8. Commit is sent with actual amount and optional metrics
9. Heartbeat is cancelled
10. If method threw: reservation is released instead of committed

## Summary

The Cycles Spring Boot Starter turns budget enforcement into a single annotation:

- `@Cycles("estimate")` wraps any method in a reserve → execute → commit lifecycle
- SpEL expressions provide dynamic cost estimation
- Heartbeat extensions keep reservations alive for long-running operations
- Caps, metrics, and metadata are accessible through `CyclesContextHolder`
- DENY decisions throw catchable exceptions for degradation handling
- Durable settlement recovery preserves known actual usage across ambiguous outcomes and JVM restarts

This gives Spring applications production-grade budget enforcement with minimal code changes.

## Next steps

To explore the Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)



# Getting Started with the Cycles MCP Server

[![npm downloads](https://img.shields.io/npm/dt/@runcycles/mcp-server?label=MCP%20Server%20downloads&color=555&style=flat-square)](https://www.npmjs.com/package/@runcycles/mcp-server)

The Cycles MCP Server gives MCP-compatible agents access to Cycles runtime authority tools: reserve, commit, release, decide, check balance, and record events. Instead of integrating an SDK into your application code, you add the MCP server to your agent's tool configuration and the agent gets direct access to those tools.

This is the fastest way to expose Cycles budget tools to an MCP-compatible AI agent. For hard production enforcement, route costly or risky actions through the reserve → execute → commit/release lifecycle, or enforce Cycles in the application/gateway layer.

::: warning What this does and does not enforce
The MCP server **exposes Cycles tools** to the agent. It does not automatically proxy or block every other MCP tool, API call, or model request — the agent can still take actions that bypass Cycles unless the host requires a live `cycles_reserve` before execution and settles the reservation afterward. `cycles_decide` is a non-locking preflight check.

Use this for:
- budget-aware agents and operator workflows
- explicit reserve / commit / release flows
- demos and local integration

For deterministic production enforcement, make the Cycles check part of the tool execution path itself — at the SDK, gateway, or framework adapter layer.
:::

::: tip Cycles provides three runtime-authority pillars
- **Spend** — `cycles_reserve` / `cycles_commit` / `cycles_release` enforce budget before instrumented agent actions
- **Risky actions** — `cycles_decide` performs a non-locking preflight using caller-assigned `RISK_POINTS` estimates and can return `ALLOW`, `ALLOW_WITH_CAPS`, or `DENY`. The application must apply the decision and any returned caps.
- **Audit** — `cycles_create_event` and reserve-commit lifecycle calls create structured records for export, compliance, attribution, and incident review
:::

## Prerequisites

- **A running Cycles stack** with a tenant, API key, and budget. If you don't have one yet, follow [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack) first.
- **Node.js 20+ with `npx` available** for Claude Code, Cursor, Windsurf, or a manual Claude Desktop configuration. The recommended Claude Desktop `.mcpb` extension uses Claude Desktop's bundled runtime and does not require a separate Node.js installation.

::: tip Where do I get my API key?
API keys are created through the **Cycles Admin Server** (port 7979). Use a runtime API key such as `cyc_live_...`. If your stack is already running with a tenant, create one directly:

```bash
curl -s -X POST http://localhost:7979/v1/admin/api-keys \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: admin-bootstrap-key" \
  -d '{
    "tenant_id": "acme-corp",
    "name": "mcp-key",
    "permissions": ["reservations:create","reservations:commit","reservations:release","reservations:extend","reservations:list","balances:read"]
  }' | jq -r '.key_secret'
```

The response returns the full key (e.g. `cyc_live_abc123...`). **Save it — the secret is only shown once.**

The permissions above are the valid runtime permissions used by the MCP tool set. The current permission schema does not define separate `decide` or `events:create` values; including either causes API-key creation to fail validation. For a least-privilege key, omit read or lifecycle permissions for tools the agent does not need.

Need the full setup? See [Deploy the Full Stack — Create an API key](/quickstart/deploying-the-full-cycles-stack#step-3-create-an-api-key). For rotation and lifecycle details, see [API Key Management](/how-to/api-key-management-in-cycles).
:::

## Pick your client

Each client has its own config file path and quirks. Start with the one you use:

| Client | Quickstart |
|---|---|
| **Claude Desktop** | [Add Cycles to Claude Desktop](/quickstart/mcp-claude-desktop) |
| **Claude Code** | [Add Cycles to Claude Code](/quickstart/mcp-claude-code) |
| **Cursor** | [Add Cycles to Cursor](/quickstart/mcp-cursor) |
| **Windsurf** | [Add Cycles to Windsurf](/quickstart/mcp-windsurf) |
| Other MCP-compatible client | Use the STDIO config below as a template |

All hosts use the same `@runcycles/mcp-server` implementation. Claude Desktop can install the bundled `.mcpb` desktop extension; the other local setup paths launch the npm package via `npx`. Config-file paths and client-specific behavior still differ.

### Generic STDIO config (template)

```json
{
  "mcpServers": {
    "cycles": {
      "command": "npx",
      "args": ["-y", "@runcycles/mcp-server"],
      "env": {
        "CYCLES_API_KEY": "cyc_live_...",
        "CYCLES_BASE_URL": "http://localhost:7878"
      }
    }
  }
}
```

### Mock mode (no backend required)

To try the server without a running Cycles stack, set `CYCLES_MOCK: "true"` instead of the API key / base URL. Mock mode returns realistic synthetic responses; generated IDs and timestamps vary between calls. It performs no live enforcement. In a production Node environment, the server refuses to start in mock mode unless `CYCLES_ALLOW_MOCK_IN_PRODUCTION` is explicitly set to `"true"`.

```json
{
  "mcpServers": {
    "cycles": {
      "command": "npx",
      "args": ["-y", "@runcycles/mcp-server"],
      "env": { "CYCLES_MOCK": "true" }
    }
  }
}
```

### Running the server over HTTP

For a shared remote MCP gateway (multi-developer team, cloud deploy, sidecar in CI), see [Running the MCP server over HTTP](/how-to/running-the-mcp-server-over-http). STDIO is the right default for a single developer on a local machine.

## Your first budget check

Once connected, ask your agent to check a budget balance:

> "Check the budget balance for tenant acme-corp"

The agent will call `cycles_check_balance` with `tenant: "acme-corp"` and return matching balance records — remaining budget, reserved amounts, and total spent. If you need descendant scopes, ask for child scopes explicitly; the tool maps that to `includeChildren: true` where the server supports it.

## The reserve-commit lifecycle

The core pattern is **reserve → execute → commit**. Commit the actual usage whenever execution incurred cost, including an operation that started but later failed. Use **release** only when the reservation was unused — for example, the operation was cancelled, skipped, or failed before execution. Here's how it works through MCP tools:

**Step 1 — Reserve** before doing something expensive:

> "Reserve 500,000 USD_MICROCENTS for an OpenAI GPT-4o call"

The agent calls `cycles_reserve` and gets back a `reservationId` and a decision of `ALLOW` or `ALLOW_WITH_CAPS`. The budget is locked and the agent can proceed, applying any returned caps. If a live reservation is rejected, the tool returns an error such as `BUDGET_EXCEEDED`; `decision: "DENY"` is returned only by `cycles_decide` or a reserve call with `dryRun: true`.

**Step 2 — Execute** the operation (the LLM call, API request, etc.)

**Step 3 — Commit** actual usage:

> "Commit reservation res_abc123 with actual usage 423,100 USD_MICROCENTS"

The agent calls `cycles_commit` with the `reservationId` and the actual amount. The difference between the reserved estimate and the actual usage is returned to the budget pool.

If no execution or billable work occurred, the agent calls `cycles_release` instead to return the full reserved amount. If the operation incurred partial usage before failing, commit that actual usage rather than releasing the reservation.

## Handling decisions

`cycles_decide` and `cycles_reserve` with `dryRun: true` can return any of the three decisions below. A successful live `cycles_reserve` returns `ALLOW` or `ALLOW_WITH_CAPS`; a live denial is surfaced as an MCP tool error carrying the Cycles error code and HTTP status.

| Decision | Meaning | Agent should… |
|----------|---------|---------------|
| `ALLOW` | Budget is available, proceed normally | Execute the operation |
| `ALLOW_WITH_CAPS` | The deepest matching budget has configured caps | Apply the returned constraints before execution. The `caps` field can contain `maxTokens`, `maxStepsRemaining`, `toolAllowlist`, `toolDenylist`, and `cooldownMs` |
| `DENY` | Budget exhausted or insufficient | Stop, inform the user, or switch to a free fallback |

## Available tools

The MCP server exposes 9 tools:

| Tool | Description |
|------|-------------|
| `cycles_reserve` | Reserve budget before a costly operation. Returns a reservation ID and decision |
| `cycles_commit` | Commit actual usage after an operation completes. Records actual usage against the budget |
| `cycles_release` | Release a reservation without committing. Returns budget to the pool |
| `cycles_extend` | Extend the TTL of an active reservation (heartbeat for long-running ops) |
| `cycles_decide` | Lightweight preflight check — ask if an action would be allowed without reserving |
| `cycles_check_balance` | Check current budget balance for a scope |
| `cycles_list_reservations` | List reservations, filtered by status or subject |
| `cycles_get_reservation` | Get details of a specific reservation by ID |
| `cycles_create_event` | Record completed usage directly without a reservation lifecycle. This is post-hoc direct-debit metering, not arbitrary governance-event ingestion or pre-execution enforcement |

## Built-in prompts

The server includes 3 prompts that agents can invoke for guided workflows:

| Prompt | Description |
|--------|-------------|
| `integrate_cycles` | Generate reserve-commit lifecycle patterns for a specific language and use case |
| `diagnose_overrun` | Analyze budget exhaustion — guides through checking balances and listing reservations |
| `design_budget_strategy` | Recommend scope hierarchy, limits, units, and degradation strategy for a workflow |

## Configuration reference

| Variable | Default | Description |
|----------|---------|-------------|
| `CYCLES_API_KEY` | *(required in real mode)* | API key for authenticating with the Cycles server |
| `CYCLES_BASE_URL` | *(required in real mode)* | Base URL of your Cycles server (e.g., `http://localhost:7878`) |
| `CYCLES_MOCK` | — | Set to `"true"` to use mock mode (no server needed) |
| `CYCLES_ALLOW_MOCK_IN_PRODUCTION` | `false` | Must be `"true"` to allow mock mode when `NODE_ENV=production`; use only intentionally because mock mode disables enforcement |
| `CYCLES_DEFAULT_TENANT` | — | Default `subject.tenant` when the caller omits it |
| `CYCLES_DEFAULT_WORKSPACE` | — | Default `subject.workspace` when the caller omits it |
| `CYCLES_DEFAULT_APP` | — | Default `subject.app` when the caller omits it |
| `CYCLES_DEFAULT_WORKFLOW` | — | Default `subject.workflow` when the caller omits it |
| `CYCLES_DEFAULT_AGENT` | — | Default `subject.agent` when the caller omits it |
| `CYCLES_DEFAULT_TOOLSET` | — | Default `subject.toolset` when the caller omits it |
| `PORT` | `3000` | HTTP port when using `--transport http` |
| `HOST` | all interfaces | HTTP bind address; set `127.0.0.1` for loopback-only access |
| `MCP_HTTP_AUTH_TOKEN` | — | Shared bearer token required on every `/mcp` request when configured; `/health` remains public |

Explicit subject fields always override `CYCLES_DEFAULT_*` values, and custom `dimensions` are never defaulted. Defaults apply to `cycles_reserve`, `cycles_decide`, `cycles_create_event`, and `cycles_check_balance`; they do not rewrite existing reservations or reservation-list filters. Blank defaults are ignored, while whitespace-only or over-128-character values fail validation.

Every mutating tool still requires a caller-supplied `idempotencyKey`. Reuse the same key when retrying the same logical operation so Cycles can deduplicate the request. Under budget pressure, tool responses may also append plain-text agent hints after the structured JSON for `DENY`, `ALLOW_WITH_CAPS`, or balances below roughly 15% remaining.

## Next steps

- **[Integrating Cycles with MCP](/how-to/integrating-cycles-with-mcp)** — advanced patterns: preflight decisions, graceful degradation, long-running operations, fire-and-forget events
- **[Running the MCP server over HTTP](/how-to/running-the-mcp-server-over-http)** — when to use HTTP transport, and how to deploy a shared remote MCP gateway
- **[Architecture Overview](/quickstart/architecture-overview-how-cycles-fits-together)** — how the MCP server fits into the full Cycles stack
- **[End-to-End Tutorial](/quickstart/end-to-end-tutorial)** — walk through the complete reserve → commit lifecycle hands-on
- **[Cost Estimation Cheat Sheet](/how-to/cost-estimation-cheat-sheet)** — estimate token costs for popular LLM models



# Getting Started with the Python Client

[![PyPI downloads](https://img.shields.io/pypi/dm/runcycles?label=PyPI%20downloads&color=555&style=flat-square)](https://pypi.org/project/runcycles/)

The `runcycles` Python package provides both a `@cycles` decorator and a programmatic `CyclesClient` for adding budget enforcement to any Python application.

The decorator wraps any function in a reserve → execute → commit lifecycle:

1. **Before the function runs:** evaluates the estimate, creates a reservation, and checks the decision
2. **While the function runs:** maintains the reservation with automatic heartbeat extensions
3. **After the function returns:** commits actual usage and releases any unused remainder
4. **If the function raises:** releases the reservation to return budget to the pool

Once actual usage is known, the current client persists settlement before the first commit request. Ambiguous outcomes replay with the same key, and an expired commit is recovered through `POST /v1/events`. The guarantee cannot cover a process death before actual usage is known; see [SDK Settlement Recovery and Durability](/protocol/sdk-settlement-recovery-and-durability).

::: tip Cycles provides three runtime-authority pillars
- **Spend** — reserve-commit budget enforcement before instrumented LLM calls and tool actions
- **Risky actions** — callers can budget assigned `RISK_POINTS`; applications must apply preflight decisions and any configured caps
- **Audit** — reservations, commits, releases, and direct-usage events create lifecycle records; non-persisting preflight decisions need application logging
:::

## Prerequisites

You need a running Cycles stack with a tenant, API key, and budget. If you don't have one yet, follow [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack) first.

::: tip Where do I get my API key?
API keys are created through the **Cycles Admin Server** (port 7979) and always start with `cyc_live_`. If your stack is already running with a tenant, create one directly:

```bash
curl -s -X POST http://localhost:7979/v1/admin/api-keys \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: admin-bootstrap-key" \
  -d '{
    "tenant_id": "acme-corp",
    "name": "dev-key",
    "permissions": ["reservations:create","reservations:commit","reservations:release","reservations:extend","reservations:list","balances:read"]
  }' | jq -r '.key_secret'
```

The response returns the full key (e.g. `cyc_live_abc123...`). **Save it — the secret is only shown once.**

Need the full setup? See [Deploy the Full Stack — Create an API key](/quickstart/deploying-the-full-cycles-stack#step-3-create-an-api-key). For rotation and lifecycle details, see [API Key Management](/how-to/api-key-management-in-cycles).
:::

## Verify your server is running

Before writing any code, confirm the Cycles Server is reachable:

```bash
curl -sf http://localhost:7878/actuator/health | jq .
```

You should see `{"status":"UP"}`. If this fails, check that the server is running per [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack).

::: info Two API key types
Cycles uses two different authentication headers:

- **`X-Admin-API-Key`** — used with the **Admin Server** (port 7979) to manage tenants, budgets, and API keys. This is the bootstrap secret (e.g. `admin-bootstrap-key`).
- **`X-Cycles-API-Key`** — used with the **Cycles Server** (port 7878) for runtime operations (reservations, commits, balances). This is the tenant-scoped key starting with `cyc_live_...`.

The `runcycles` client uses `X-Cycles-API-Key` automatically. You only need `X-Admin-API-Key` when calling the Admin Server directly (e.g. to create tenants or API keys).
:::

## Installation

```bash
pip install runcycles
```

Requires Python 3.10+. Dependencies (`httpx`, `pydantic >= 2.0`) are installed automatically.

## Configuration

```python
from runcycles import CyclesConfig

config = CyclesConfig(
    base_url="http://localhost:7878",
    api_key="cyc_live_...",  # from Admin Server — see tip above
    tenant="acme-corp",
)
```

Or from environment variables:

```bash
export CYCLES_BASE_URL=http://localhost:7878
export CYCLES_API_KEY=cyc_live_...   # from Admin Server /v1/admin/api-keys response
export CYCLES_TENANT=acme-corp
```

```python
config = CyclesConfig.from_env()
```

## The @cycles decorator

The simplest usage — wrap a function with a fixed estimate:

```python
from runcycles import CyclesClient, cycles, set_default_client

client = CyclesClient(config)
set_default_client(client)

@cycles(estimate=1000) # [!code focus]
def summarize(text: str) -> str:
    return call_llm(text)

result = summarize("Hello world")
```

This reserves 1000 USD_MICROCENTS before `summarize()` runs, then commits the same amount afterward.

### Dynamic estimates

The estimate can be a callable that receives the function's arguments:

```python
@cycles(estimate=lambda text, max_tokens: max_tokens * 10) # [!code focus]
def generate(text: str, max_tokens: int) -> str:
    return call_llm(text, max_tokens=max_tokens)
```

### Specifying actual usage

By default, the estimate is used as the actual amount at commit time. To calculate actual usage from the return value:

```python
@cycles(
    estimate=5000,
    actual=lambda result: len(result) * 5, # [!code focus]
)
def chat(prompt: str) -> str:
    return call_llm(prompt)
```

### Decorator parameters

| Parameter | Default | Description |
|---|---|---|
| `estimate` | (required) | `int` or callable returning `int`. Estimated amount. |
| `actual` | `None` | `int` or callable receiving the return value. Defaults to estimate. |
| `action_kind` | `None` | Action category (e.g. `"llm.completion"`). `str` or callable. |
| `action_name` | `None` | Action identifier (e.g. `"gpt-4"`). `str` or callable. |
| `action_tags` | `None` | List of tags for filtering/reporting. `list[str]` or callable. |
| `unit` | `USD_MICROCENTS` | Budget unit: `USD_MICROCENTS`, `TOKENS`, `CREDITS`, `RISK_POINTS`. |
| `ttl_ms` | `60000` | Reservation TTL in milliseconds. |
| `grace_period_ms` | `None` | Grace period after TTL expiry. When `None`, server default (5000ms) applies. |
| `overage_policy` | `"ALLOW_IF_AVAILABLE"` | `"REJECT"`, `"ALLOW_IF_AVAILABLE"`, or `"ALLOW_WITH_OVERDRAFT"`. |
| `dry_run` | `False` | If `True`, evaluate without persisting. Function does not execute. |
| `tenant` | `None` | Subject tenant override. `str` or callable. |
| `workspace` | `None` | Subject workspace override. `str` or callable. |
| `app` | `None` | Subject app override. `str` or callable. |
| `workflow` | `None` | Subject workflow override. `str` or callable. |
| `agent` | `None` | Subject agent override. `str` or callable. |
| `toolset` | `None` | Subject toolset override. `str` or callable. |
| `dimensions` | `None` | Custom dimensions dict. `dict[str, str]` or callable. |
| `client` | `None` | Explicit client. Falls back to module default. |
| `use_estimate_if_actual_not_provided` | `True` | If `True` and `actual` is `None`, use estimate as actual at commit. |

### Dynamic subject and action fields

Since 0.4.0, `action_kind`, `action_name`, `action_tags`, the six subject parameters (`tenant`, `workspace`, `app`, `workflow`, `agent`, `toolset`), and `dimensions` also accept a callable. The callable is invoked with the decorated function's `*args, **kwargs` at reservation time, so subject and action can be routed per call:

```python
@cycles(
    estimate=1000,
    workspace=lambda req, workspace_id: workspace_id, # [!code focus]
    action_kind=lambda req, *_: f"llm.{req.provider}", # [!code focus]
    action_name=lambda req, *_: req.model, # [!code focus]
)
def run_request(req: Request, workspace_id: str) -> Response:
    ...
```

A falsy result (e.g. `None`) falls through: subject fields fall back to the config default, `action_kind`/`action_name` fall back to `"unknown"`, and `action_tags`/`dimensions` are omitted from the request.

## Accessing reservation context at runtime

Inside a decorated function, the current reservation context is available via `get_cycles_context()`:

```python :line-numbers
from runcycles import cycles, get_cycles_context, CyclesMetrics

@cycles(estimate=1000)
def process(text: str) -> str:
    ctx = get_cycles_context() # [!code focus]

    # Check reservation details
    print(f"Reservation: {ctx.reservation_id}")
    print(f"Decision: {ctx.decision}")

    # Check caps (if ALLOW_WITH_CAPS)
    if ctx.has_caps():
        max_tokens = ctx.caps.max_tokens
        if not ctx.caps.is_tool_allowed("web.search"):
            pass  # skip web search

    # Attach metrics for the commit
    ctx.metrics = CyclesMetrics(
        tokens_input=150,
        tokens_output=80,
        latency_ms=320,
        model_version="gpt-4o-mini",
    )

    # Attach metadata for audit
    ctx.commit_metadata = {"app_request_id": "req-abc-123"}

    return call_llm(text)
```

## Decision handling

When the reservation decision comes back, the decorator handles each case:

- **ALLOW** — the function runs normally.
- **ALLOW_WITH_CAPS** — the function runs. Caps are available through `get_cycles_context()` for the function to inspect and respect.
- **DENY** — the function does not run. `CyclesProtocolError` is raised, with `reason_code` set. A specific subclass (e.g. `BudgetExceededError`) is raised only when the server returns a matching HTTP error code; a 200 response with `decision=DENY` raises the plain `CyclesProtocolError`.

```python
from runcycles import BudgetExceededError, CyclesProtocolError

try:
    result = summarize("Hello")
except BudgetExceededError: # [!code focus]
    result = fallback_response()
except CyclesProtocolError as e:
    if e.retry_after_ms:
        # retry after suggested delay
        pass
    result = fallback_response()
```

## Async support

The `@cycles` decorator works with async functions automatically:

```python
from runcycles import AsyncCyclesClient, cycles, set_default_client

async_client = AsyncCyclesClient(config)
set_default_client(async_client)

@cycles(estimate=1000) # [!code focus]
async def async_summarize(text: str) -> str:
    return await call_llm_async(text)

result = await async_summarize("Hello")
```

## Programmatic client

For full control, use `CyclesClient` directly:

```python :line-numbers
from runcycles import (
    CyclesClient, ReservationCreateRequest, CommitRequest, ReleaseRequest,
    Subject, Action, Amount, Unit, CyclesMetrics,
)

with CyclesClient(config) as client:
    # 1. Reserve
    response = client.create_reservation(ReservationCreateRequest( # [!code focus]
        idempotency_key="req-001",
        subject=Subject(tenant="acme", agent="support-bot"),
        action=Action(kind="llm.completion", name="gpt-4"),
        estimate=Amount(unit=Unit.USD_MICROCENTS, amount=500_000),
        ttl_ms=30_000,
    ))

    if not response.is_success:
        raise RuntimeError(f"Reservation failed: {response.error_message}")

    # Defensive: a conformant server returns 409 on live budget denial, but
    # dry-run responses (and lenient servers) return 200 with decision=DENY
    # and no reservation_id - check before using it
    if response.get_body_attribute("decision") == "DENY":
        raise RuntimeError(
            f"Reservation denied: {response.get_body_attribute('reason_code')}"
        )

    reservation_id = response.get_body_attribute("reservation_id")

    # 2. Execute
    try:
        result = call_llm("Hello")

        # 3. Commit
        client.commit_reservation(reservation_id, CommitRequest( # [!code focus]
            idempotency_key="commit-001",
            actual=Amount(unit=Unit.USD_MICROCENTS, amount=420_000),
            metrics=CyclesMetrics(tokens_input=1200, tokens_output=800),
        ))

    except Exception:
        # 4. Release on failure
        client.release_reservation(reservation_id, ReleaseRequest(
            idempotency_key="release-001",
            reason="Processing failed",
        ))
        raise
```

### Preflight decision check

```python
from runcycles import DecisionRequest

response = client.decide(DecisionRequest( # [!code focus]
    idempotency_key="decide-001",
    subject=Subject(tenant="acme"),
    action=Action(kind="llm.completion", name="gpt-4"),
    estimate=Amount(unit=Unit.USD_MICROCENTS, amount=500_000),
))

decision = response.get_body_attribute("decision")  # "ALLOW", "ALLOW_WITH_CAPS", or "DENY"
```

### Querying balances

```python
response = client.get_balances(tenant="acme")
print(response.body)
```

At least one subject filter kwarg (`tenant`, `workspace`, `app`, `workflow`, `agent`, or `toolset`) is required — calling `get_balances()` with none raises `ValueError`.

### Recording events (direct debit)

```python
from runcycles import EventCreateRequest

response = client.create_event(EventCreateRequest( # [!code focus]
    idempotency_key="evt-001",
    subject=Subject(tenant="acme"),
    action=Action(kind="api.call", name="geocode"),
    actual=Amount(unit=Unit.USD_MICROCENTS, amount=1_500),
))
```

## Suggested walkthrough

Follow this order to build understanding progressively:

**1. Reserve and commit with a fixed estimate**

```python
from runcycles import CyclesClient, CyclesConfig, cycles, set_default_client

config = CyclesConfig(base_url="http://localhost:7878", api_key="cyc_live_...", tenant="acme-corp")
client = CyclesClient(config)
set_default_client(client)

@cycles(estimate=1000) # [!code focus]
def hello(name: str) -> str:
    return f"Hello, {name}!"

result = hello("world")
print(result)
```

**2. Check your balance**

```python
response = client.get_balances(tenant="acme-corp")
print(response.body)
```

**3. Try a dry run**

```python
@cycles(estimate=500, dry_run=True)
def dry_run_func() -> str:
    return "This won't consume budget"

dry_run_func()
# Check balances — they haven't changed
```

**4. Use dynamic estimates with metrics**

```python :line-numbers
from runcycles import get_cycles_context, CyclesMetrics

@cycles(
    estimate=lambda prompt, max_tokens: max_tokens * 10, # [!code focus]
    actual=lambda result: len(result) * 5, # [!code focus]
    action_kind="llm.completion",
    action_name="gpt-4",
)
def generate(prompt: str, max_tokens: int) -> str:
    ctx = get_cycles_context()
    ctx.metrics = CyclesMetrics(tokens_input=len(prompt), tokens_output=max_tokens)
    return f"Generated response for: {prompt}"

result = generate("Explain budgets", max_tokens=500)
```

**5. Handle denials gracefully**

```python
from runcycles import BudgetExceededError

@cycles(estimate=999_999_999)
def expensive_func() -> str:
    return "This needs a lot of budget"

try:
    expensive_func()
except BudgetExceededError: # [!code focus]
    print("Budget exhausted — using fallback")
```

## Nested `@cycles` calls

Calling a `@cycles`-decorated function from inside another `@cycles`-decorated function is allowed — it will not raise an error. However, each decorator creates an **independent reservation** that deducts budget separately:

```python
@cycles(estimate=100, action_name="inner")
def inner_call():
    return "done"

@cycles(estimate=500, action_name="outer")
def outer_call():
    return inner_call()  # creates a SECOND reservation — 600 total deducted, not 500
```

This means nested decorators **double-count budget**. The outer reservation already covers the full estimated cost of the operation, so an inner reservation deducts additional budget from the same pool.

**Recommended pattern:** Place `@cycles` at the outermost entry point only. Inner functions should be plain functions without their own guard:

```python
def inner_call():  # no @cycles — called within a guarded operation
    return "done"

@cycles(estimate=500, action_name="outer")
def outer_call():
    return inner_call()  # single reservation — 500 total
```

## Lifecycle summary

For each `@cycles`-decorated function call:

1. Estimate is evaluated (callable or fixed value)
2. Reservation is created on the Cycles server
3. Decision is checked (ALLOW / ALLOW_WITH_CAPS / DENY)
4. If DENY: exception is raised, function does not run
5. Heartbeat extension is scheduled (background thread; asyncio task for async functions)
6. Function executes
7. Actual usage is evaluated (callable, fixed value, or estimate); a failing or invalid callback commits the estimate with `metadata.actual_source=estimate`
8. Commit is sent with actual amount and optional metrics
9. Heartbeat is cancelled
10. If the guarded function raised: reservation is released instead of committed
11. If post-action settlement setup failed: the error surfaces, but known spend is never released

If estimate fallback is disabled without an `actual`, the client rejects the
configuration before creating a reservation or running the function. A
recognized terminal commit rejection stops retry and discards the unrecoverable
journal record without releasing the reservation after spend occurred.

## Next steps

- [Integrating with OpenAI Agents SDK](/how-to/integrating-cycles-with-openai-agents) — budget governance for multi-agent workflows
- [Error Handling in Python](/how-to/error-handling-patterns-in-python) — Python-specific exception hierarchy and patterns
- [Error Handling Patterns](/how-to/error-handling-patterns-in-cycles-client-code) — general error handling patterns
- [API Reference](/api/) — interactive endpoint documentation
- [Using the Client Programmatically](/how-to/using-the-cycles-client-programmatically) — programmatic client reference



# Rust Client Quickstart — Budget Control for AI Agents

[![Crates.io downloads](https://img.shields.io/crates/d/runcycles?label=crates.io%20downloads&color=555&style=flat-square)](https://crates.io/crates/runcycles)

Building AI agents in Rust with Tokio? You need hard limits on LLM spending and tool-call exposure **before** they execute, not after. The Cycles Rust client (`runcycles` crate) gives any async Rust application a reserve-commit budget enforcement layer with three integration levels — from a one-line `with_cycles()` wrapper to RAII guards to a low-level programmatic client.

Same wire protocol as the [Python](/quickstart/getting-started-with-the-python-client), [TypeScript](/quickstart/getting-started-with-the-typescript-client), and [Spring Boot](/quickstart/getting-started-with-the-cycles-spring-boot-starter) clients — switch languages without changing your Cycles server.

::: tip Cycles provides three runtime-authority pillars
- **Spend** — reserve-commit budget enforcement before instrumented LLM calls and tool actions
- **Risky actions** — callers can budget assigned `RISK_POINTS`; applications must apply preflight decisions and any configured caps
- **Audit** — reservations, commits, releases, and direct-usage events create lifecycle records; non-persisting preflight decisions need application logging
:::

The `runcycles` crate provides three levels of budget enforcement for any async Rust application:

1. **`with_cycles()`** — automatic reserve → execute → commit/release (like Python's `@cycles` decorator)
2. **`ReservationGuard`** — RAII guard for manual control (streaming, multi-step workflows)
3. **`CyclesClient`** — low-level programmatic API for full control

The two high-level integrations own this lifecycle; the low-level client exposes the same protocol operations for you to compose:

1. **Before the operation:** evaluates the estimate, creates a reservation, and checks the decision
2. **While the operation runs:** maintains the reservation with automatic heartbeat extensions
3. **After the operation returns:** commits actual usage and releases any unused remainder
4. **If the operation fails:** releases the reservation to return budget to the pool

## Prerequisites

You need a running Cycles stack with a tenant, API key, and budget. If you don't have one yet, follow [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack) first.

::: tip Where do I get my API key?
API keys are created through the **Cycles Admin Server** (port 7979) and always start with `cyc_live_`. If your stack is already running with a tenant, create one directly:

```bash
curl -s -X POST http://localhost:7979/v1/admin/api-keys \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: admin-bootstrap-key" \
  -d '{
    "tenant_id": "acme-corp",
    "name": "dev-key",
    "permissions": ["reservations:create","reservations:commit","reservations:release","reservations:extend","reservations:list","balances:read"]
  }' | jq -r '.key_secret'
```

The response returns the full key (e.g. `cyc_live_abc123...`). **Save it — the secret is only shown once.**

Need the full setup? See [Deploy the Full Stack — Create an API key](/quickstart/deploying-the-full-cycles-stack#step-3-create-an-api-key). For rotation and lifecycle details, see [API Key Management](/how-to/api-key-management-in-cycles).
:::

## Verify your server is running

Before writing any code, confirm the Cycles Server is reachable:

```bash
curl -sf http://localhost:7878/actuator/health | jq .
```

You should see `{"status":"UP"}`. If this fails, check that the server is running per [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack).

::: info Two API key types
Cycles uses two different authentication headers:

- **`X-Admin-API-Key`** — used with the **Admin Server** (port 7979) to manage tenants, budgets, and API keys. This is the bootstrap secret (e.g. `admin-bootstrap-key`).
- **`X-Cycles-API-Key`** — used with the **Cycles Server** (port 7878) for runtime operations (reservations, commits, balances). This is the tenant-scoped key starting with `cyc_live_...`.

The `runcycles` client uses `X-Cycles-API-Key` automatically. You only need `X-Admin-API-Key` when calling the Admin Server directly (e.g. to create tenants or API keys).
:::

## Installation

```bash
cargo add runcycles
```

Or add to `Cargo.toml`:

```toml
[dependencies]
runcycles = "0.3"
tokio = { version = "1", features = ["full"] }
```

Requires Rust 1.88+. Dependencies (`reqwest`, `serde`, `tokio`) are installed automatically.

`ReservationGuard::commit()` persists known actual usage before its first settlement request. Ambiguous outcomes remain queued with the original idempotency key, and expired commits recover through `POST /v1/events`. See [SDK Settlement Recovery and Durability](/protocol/sdk-settlement-recovery-and-durability).

## Configuration

```rust
use runcycles::CyclesClient;

let client = CyclesClient::builder(
    "cyc_live_...",        // from Admin Server — see tip above
    "http://localhost:7878",
)
.tenant("acme-corp")
.build();
```

Or from environment variables:

```bash
export CYCLES_BASE_URL=http://localhost:7878
export CYCLES_API_KEY=cyc_live_...   # from Admin Server /v1/admin/api-keys response
export CYCLES_TENANT=acme-corp
```

```rust
let config = CyclesConfig::from_env().expect("missing CYCLES_ env vars");
let client = CyclesClient::new(config);
```

For the complete configuration surface — retry tuning, custom `reqwest::Client`, blocking variant, env var prefix customization — see the [Rust Client Configuration Reference](/configuration/rust-client-configuration-reference).

## Automatic lifecycle with `with_cycles()`

The simplest way to add budget enforcement — wrap any async operation:

```rust
use runcycles::{CyclesClient, with_cycles, WithCyclesConfig, models::*};

let reply = with_cycles( // [!code focus]
    &client,
    WithCyclesConfig::new(Amount::tokens(1000))
        .action("llm.completion", "gpt-4o")
        .subject(Subject { tenant: Some("acme-corp".into()), ..Default::default() }),
    |ctx| async move { // [!code focus]
        // ctx.caps, ctx.decision, ctx.reservation_id available
        let response = call_llm("Hello").await;
        let actual_tokens = 42; // from your LLM response usage stats
        Ok((response, Amount::tokens(actual_tokens))) // [!code focus]
    },
).await?;
// On success → auto-commits actual_tokens. On error → auto-releases.
```

The closure returns `Ok((result, actual_cost))` — a tuple of your return value and the actual `Amount` spent. If the closure returns `Err`, the reservation is released automatically.

### `WithCyclesConfig` parameters

| Parameter | Default | Description |
|---|---|---|
| `new(estimate)` | (required) | `Amount` — estimated cost to reserve |
| `.action(kind, name)` | `"unknown"` | Action category and identifier (e.g. `"llm.completion"`, `"gpt-4o"`) |
| `.subject(subject)` | `Default` | Who is spending (tenant, workspace, app, etc.) |
| `.ttl_ms(ms)` | `60000` | Reservation TTL in milliseconds |
| `.grace_period_ms(ms)` | server default | Grace period after TTL expiry |
| `.overage_policy(policy)` | server default | `Reject`, `AllowIfAvailable`, or `AllowWithOverdraft` |
| `.action_tags(tags)` | `None` | Tags for filtering/reporting |
| `.metrics(metrics)` | `None` | Attach observability metrics to the commit |

### Accessing context inside the closure

The closure receives a `GuardContext` with the reservation state:

```rust
|ctx| async move {
    // Budget decision
    println!("Decision: {:?}", ctx.decision);       // Allow or AllowWithCaps
    println!("Reservation: {}", ctx.reservation_id);

    // Check caps (if ALLOW_WITH_CAPS)
    if let Some(caps) = &ctx.caps { // [!code focus]
        let max_tokens = caps.max_tokens.unwrap_or(1000); // [!code focus]
        if !caps.is_tool_allowed("web_search") {
            // skip web search — budget policy restricts it
        }
    }

    let result = call_llm("Hello").await;
    Ok((result, Amount::tokens(42)))
}
```

## RAII guard for manual control

For streaming, multi-step workflows, or when you need full control over when to commit:

```rust
use runcycles::{CyclesClient, models::*};

// 1. Reserve
let guard = client.reserve( // [!code focus]
    ReservationCreateRequest::builder()
        .subject(Subject { tenant: Some("acme-corp".into()), ..Default::default() })
        .action(Action::new("llm.completion", "gpt-4o"))
        .estimate(Amount::tokens(2000))
        .ttl_ms(30_000_u64)
        .build()
).await?;

// 2. Check caps
if let Some(caps) = guard.caps() {
    println!("Max tokens: {:?}", caps.max_tokens);
}

// 3. Execute (e.g. stream chunks, accumulate tokens)
let mut total_tokens = 0i64;
for chunk in stream_llm("Write a poem").await {
    total_tokens += chunk.tokens;
}

// 4. Commit — consumes the guard (double-commit = compile error) // [!code focus]
guard.commit( // [!code focus]
    CommitRequest::builder()
        .actual(Amount::tokens(total_tokens))
        .metrics(CyclesMetrics {
            tokens_input: Some(100),
            tokens_output: Some(total_tokens - 100),
            ..Default::default()
        })
        .build()
).await?;

// guard.commit(...) here would be a COMPILE ERROR
```

### Guard lifecycle

- **`guard.commit(self)`** — consumes the guard, commits actual spend. Compile error to call twice.
- **`guard.release(self, reason)`** — consumes the guard, returns budget. Use on error.
- **`guard.extend(ms)`** — manually extend TTL (normally automatic via heartbeat).
- **Drop without commit/release** — logs a warning and spawns a best-effort release.

## Low-level programmatic client

For full control over individual API calls:

```rust
use runcycles::{CyclesClient, models::*};

// Create reservation
let resp = client.create_reservation( // [!code focus]
    &ReservationCreateRequest::builder()
        .subject(Subject { tenant: Some("acme-corp".into()), ..Default::default() })
        .action(Action::new("llm.completion", "gpt-4o"))
        .estimate(Amount::usd_microcents(500_000))
        .ttl_ms(30_000_u64)
        .build()
).await?;

let reservation_id = resp.reservation_id.unwrap();

// Execute your operation...

// Commit
client.commit_reservation(&reservation_id, // [!code focus]
    &CommitRequest::builder()
        .actual(Amount::usd_microcents(420_000))
        .metrics(CyclesMetrics {
            tokens_input: Some(1200),
            tokens_output: Some(800),
            ..Default::default()
        })
        .build()
).await?;
```

### Preflight decision check

```rust
let resp = client.decide( // [!code focus]
    &DecisionRequest::builder()
        .subject(Subject { tenant: Some("acme-corp".into()), ..Default::default() })
        .action(Action::new("llm.completion", "gpt-4o"))
        .estimate(Amount::usd_microcents(500_000))
        .build()
).await?;

println!("Decision: {:?}", resp.decision); // Allow, AllowWithCaps, or Deny
```

### Querying balances

```rust
let resp = client.get_balances(&BalanceParams {
    tenant: Some("acme-corp".into()),
    ..Default::default()
}).await?;

for balance in &resp.balances {
    println!("{}: {} remaining", balance.scope, balance.remaining.amount);
}
```

### Recording events (direct debit)

```rust
let resp = client.create_event( // [!code focus]
    &EventCreateRequest::builder()
        .subject(Subject { tenant: Some("acme-corp".into()), ..Default::default() })
        .action(Action::new("api.call", "geocode"))
        .actual(Amount::usd_microcents(1_500))
        .build()
).await?;
```

## Decision handling

When the reservation decision comes back, each API level handles it:

- **`with_cycles()`** — returns `Err(Error::BudgetExceeded)` on DENY. Closure never runs.
- **`ReservationGuard`** — `client.reserve()` returns `Err(Error::BudgetExceeded)` on DENY.
- **Low-level** — `resp.decision` can be checked directly.

```rust
use runcycles::Error;

match client.reserve(/* ... */).await {
    Ok(guard) => {
        // ALLOW or ALLOW_WITH_CAPS — proceed
        guard.commit(/* ... */).await?;
    }
    Err(Error::BudgetExceeded { message, retry_after, .. }) => { // [!code focus]
        println!("Budget exceeded: {message}");
        if let Some(delay) = retry_after {
            tokio::time::sleep(delay).await;
            // retry...
        }
    }
    Err(Error::Api { status, code, .. }) => {
        println!("API error ({status}): {code:?}");
    }
    Err(Error::Transport(e)) => {
        println!("Network error (retryable): {e}");
    }
    Err(e) => {
        println!("Other error: {e}");
    }
}
```

## Suggested walkthrough

Follow this order to build understanding progressively:

**1. Reserve and commit with `with_cycles()`**

```rust
use runcycles::{CyclesClient, with_cycles, WithCyclesConfig, models::*};

let client = CyclesClient::builder("cyc_live_...", "http://localhost:7878")
    .tenant("acme-corp")
    .build();

let result = with_cycles( // [!code focus]
    &client,
    WithCyclesConfig::new(Amount::tokens(1000))
        .action("llm.completion", "gpt-4o")
        .subject(Subject { tenant: Some("acme-corp".into()), ..Default::default() }),
    |_ctx| async move {
        Ok(("Hello!".to_string(), Amount::tokens(42)))
    },
).await?;

println!("{result}");
```

**2. Check your balance**

```rust
let resp = client.get_balances(&BalanceParams {
    tenant: Some("acme-corp".into()),
    ..Default::default()
}).await?;
println!("{:?}", resp.balances);
```

**3. Try a dry run**

```rust
let resp = client.create_reservation(
    &ReservationCreateRequest::builder()
        .subject(Subject { tenant: Some("acme-corp".into()), ..Default::default() })
        .action(Action::new("llm.completion", "gpt-4o"))
        .estimate(Amount::tokens(500))
        .dry_run(true) // [!code focus]
        .build()
).await?;
println!("Decision: {:?}", resp.decision);
// Check balances — they haven't changed
```

**4. Use the RAII guard with caps**

```rust
let guard = client.reserve(/* ... */).await?;

if guard.is_capped() { // [!code focus]
    let caps = guard.caps().unwrap();
    println!("Max tokens: {:?}", caps.max_tokens);
}

guard.commit(CommitRequest::builder().actual(Amount::tokens(100)).build()).await?;
```

**5. Handle denials gracefully**

```rust
use runcycles::Error;

match with_cycles(&client, WithCyclesConfig::new(Amount::tokens(999_999_999))
    .action("llm.completion", "gpt-4o")
    .subject(Subject { tenant: Some("acme-corp".into()), ..Default::default() }),
    |_ctx| async move { Ok(("".to_string(), Amount::tokens(0))) },
).await {
    Ok(_) => println!("Success"),
    Err(Error::BudgetExceeded { message, .. }) => { // [!code focus]
        println!("Budget exhausted: {message} — using fallback");
    }
    Err(e) => println!("Error: {e}"),
}
```

## Lifecycle summary

For each `with_cycles()` call or `ReservationGuard`:

1. Estimate is provided via `Amount`
2. Reservation is created on the Cycles server
3. Decision is checked (ALLOW / ALLOW_WITH_CAPS / DENY)
4. If DENY: `Error::BudgetExceeded` is returned, operation does not run
5. Heartbeat extension is scheduled from server-authoritative `remaining_ttl_ms` when present, with a best-effort fallback for older servers
6. Operation executes
7. On success: known actual usage is journaled, then commit is sent with optional metrics
8. On error: reservation is released to return budget
9. Heartbeat is cancelled
10. If guard is dropped without commit/release: best-effort release via `tokio::spawn`

## Next steps

- [Integrate Cycles with async-openai (Rust)](/how-to/integrating-cycles-with-async-openai) — replaces the `call_llm()` placeholders above with a real OpenAI chat completion, including streaming
- [Rust Client Configuration Reference](/configuration/rust-client-configuration-reference) — full config surface, retry tuning, custom `reqwest::Client`, blocking variant
- [Error Handling in Rust](/how-to/error-handling-patterns-in-rust) — Rust-specific error patterns, retries, RAII safety, graceful degradation
- [Integrating Cycles with Rust](/how-to/integrating-cycles-with-rust) — broader integration patterns (multi-step flows, framework middleware)
- [API Reference](/api/) — interactive endpoint documentation
- [How Reserve-Commit Works](/protocol/how-reserve-commit-works-in-cycles) — the core protocol lifecycle



# Getting Started with the TypeScript Client

[![npm downloads](https://img.shields.io/npm/dt/runcycles?label=npm%20downloads&color=555&style=flat-square)](https://www.npmjs.com/package/runcycles)

The `runcycles` TypeScript package provides a `withCycles` higher-order function, a `reserveForStream` streaming adapter, and a programmatic `CyclesClient` for adding budget enforcement to any Node.js application.

The `withCycles` HOF wraps any async function in a reserve → execute → commit lifecycle:

1. **Before the function runs:** evaluates the estimate, creates a reservation, and checks the decision
2. **While the function runs:** maintains the reservation with automatic heartbeat extensions
3. **After the function returns:** commits actual usage and releases any unused remainder
4. **If the function throws:** releases the reservation to return budget to the pool

Once actual usage is known, the current client persists settlement before the first commit request. Ambiguous outcomes replay with the same key, and an expired commit is recovered through `POST /v1/events`. The guarantee cannot cover a process death before actual usage is known; see [SDK Settlement Recovery and Durability](/protocol/sdk-settlement-recovery-and-durability).

::: tip Cycles provides three runtime-authority pillars
- **Spend** — reserve-commit budget enforcement before instrumented LLM calls and tool actions
- **Risky actions** — callers can budget assigned `RISK_POINTS`; applications must apply preflight decisions and any configured caps
- **Audit** — reservations, commits, releases, and direct-usage events create lifecycle records; non-persisting preflight decisions need application logging
:::

## Prerequisites

You need a running Cycles stack with a tenant, API key, and budget. If you don't have one yet, follow [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack) first.

::: tip Where do I get my API key?
API keys are created through the **Cycles Admin Server** (port 7979) and always start with `cyc_live_`. If your stack is already running with a tenant, create one directly:

```bash
curl -s -X POST http://localhost:7979/v1/admin/api-keys \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: admin-bootstrap-key" \
  -d '{
    "tenant_id": "acme-corp",
    "name": "dev-key",
    "permissions": ["reservations:create","reservations:commit","reservations:release","reservations:extend","reservations:list","balances:read"]
  }' | jq -r '.key_secret'
```

The response returns the full key (e.g. `cyc_live_abc123...`). **Save it — the secret is only shown once.**

Need the full setup? See [Deploy the Full Stack — Create an API key](/quickstart/deploying-the-full-cycles-stack#step-3-create-an-api-key). For rotation and lifecycle details, see [API Key Management](/how-to/api-key-management-in-cycles).
:::

## Verify your server is running

Before writing any code, confirm the Cycles Server is reachable:

```bash
curl -sf http://localhost:7878/actuator/health | jq .
```

You should see `{"status":"UP"}`. If this fails, check that the server is running per [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack).

::: info Two API key types
Cycles uses two different authentication headers:

- **`X-Admin-API-Key`** — used with the **Admin Server** (port 7979) to manage tenants, budgets, and API keys. This is the bootstrap secret (e.g. `admin-bootstrap-key`).
- **`X-Cycles-API-Key`** — used with the **Cycles Server** (port 7878) for runtime operations (reservations, commits, balances). This is the tenant-scoped key starting with `cyc_live_...`.

The `runcycles` client uses `X-Cycles-API-Key` automatically. You only need `X-Admin-API-Key` when calling the Admin Server directly (e.g. to create tenants or API keys).
:::

## Installation

```bash
npm install runcycles
```

Requires Node.js 20+. TypeScript 5+ is recommended but optional — the package works with plain JavaScript. Zero runtime dependencies (uses built-in `fetch` and `AsyncLocalStorage`).

## Configuration

```typescript
import { CyclesConfig } from "runcycles";

const config = new CyclesConfig({
  baseUrl: "http://localhost:7878",
  apiKey: "cyc_live_...",  // from Admin Server — see tip above
  tenant: "acme-corp",
});
```

Or from environment variables:

```bash
export CYCLES_BASE_URL=http://localhost:7878
export CYCLES_API_KEY=cyc_live_...   # from Admin Server /v1/admin/api-keys response
export CYCLES_TENANT=acme-corp
```

```typescript
const config = CyclesConfig.fromEnv();
```

## The withCycles higher-order function

The simplest usage — wrap an async function with a fixed estimate:

```typescript :line-numbers
import { CyclesClient, CyclesConfig, withCycles, setDefaultClient } from "runcycles";

const config = new CyclesConfig({
  baseUrl: "http://localhost:7878",
  apiKey: "cyc_live_...",
  tenant: "acme-corp",
});
const client = new CyclesClient(config);
setDefaultClient(client);

const summarize = withCycles( // [!code focus]
  { estimate: 1000 }, // [!code focus]
  async (text: string) => {
    return await callLlm(text);
  },
);

const result = await summarize("Hello world");
```

This reserves 1000 USD_MICROCENTS before `summarize()` runs, then commits the same amount afterward.

### Dynamic estimates

The estimate can be a function that receives the wrapped function's arguments:

```typescript
const generate = withCycles(
  { estimate: (text: string, maxTokens: number) => maxTokens * 10 }, // [!code focus]
  async (text: string, maxTokens: number) => {
    return await callLlm(text, maxTokens);
  },
);
```

### Specifying actual usage

By default, the estimate is used as the actual amount at commit time. To calculate actual usage from the return value:

```typescript
const chat = withCycles(
  {
    estimate: 5000,
    actual: (result: string) => result.length * 5, // [!code focus]
  },
  async (prompt: string) => {
    return await callLlm(prompt);
  },
);
```

### withCycles parameters

| Parameter | Default | Description |
|---|---|---|
| `estimate` | (required) | `number` or function returning `number`. Estimated amount. |
| `actual` | `undefined` | `number` or function receiving the return value. Defaults to estimate. |
| `actionKind` | `"unknown"` | Action category (e.g. `"llm.completion"`). |
| `actionName` | `"unknown"` | Action identifier (e.g. `"gpt-4"`). |
| `actionTags` | `undefined` | Array of tags for filtering/reporting. |
| `unit` | `"USD_MICROCENTS"` | Budget unit: `"USD_MICROCENTS"`, `"TOKENS"`, `"CREDITS"`, `"RISK_POINTS"`. |
| `ttlMs` | `60000` | Reservation TTL in milliseconds (range: 1000–86400000). |
| `gracePeriodMs` | `undefined` | Grace period after TTL expiry (range: 0–60000). |
| `overagePolicy` | `"ALLOW_IF_AVAILABLE"` | `"REJECT"`, `"ALLOW_IF_AVAILABLE"`, or `"ALLOW_WITH_OVERDRAFT"`. |
| `dryRun` | `false` | If `true`, evaluate without persisting. Function does not execute. |
| `tenant` | `undefined` | Subject tenant override. |
| `workspace` | `undefined` | Subject workspace override. |
| `app` | `undefined` | Subject app override. |
| `workflow` | `undefined` | Subject workflow override. |
| `agent` | `undefined` | Subject agent override. |
| `toolset` | `undefined` | Subject toolset override. |
| `dimensions` | `undefined` | Custom dimensions object. |
| `client` | `undefined` | Explicit client. Falls back to module default. |
| `useEstimateIfActualNotProvided` | `true` | If `true` and `actual` is not set, use estimate as actual at commit. |

Since 0.3.0, `actionKind`, `actionName`, and the six subject fields (`tenant`, `workspace`, `app`, `workflow`, `agent`, `toolset`) accept `string | ((...args) => string | undefined)` — a callable is resolved per call from the wrapped function's arguments.

## Accessing reservation context at runtime

Inside a `withCycles`-guarded function, the current reservation context is available via `getCyclesContext()`:

```typescript :line-numbers
import { withCycles, getCyclesContext } from "runcycles";

const process = withCycles(
  { estimate: 1000, client },
  async (text: string) => {
    const ctx = getCyclesContext(); // [!code focus]

    // Check reservation details
    console.log(`Reservation: ${ctx?.reservationId}`);
    console.log(`Decision: ${ctx?.decision}`);

    // Check caps (if ALLOW_WITH_CAPS)
    if (ctx?.caps) {
      const maxTokens = ctx.caps.maxTokens;
      // Adjust behavior based on caps
    }

    // Attach metrics for the commit
    if (ctx) {
      ctx.metrics = {
        tokensInput: 150,
        tokensOutput: 80,
        latencyMs: 320,
        modelVersion: "gpt-4o-mini",
      };

      // Attach metadata for audit
      ctx.commitMetadata = { requestId: "req-abc-123" };
    }

    return await callLlm(text);
  },
);
```

The context uses `AsyncLocalStorage`, so it is available in any nested async call within the guarded function.

## Decision handling

When the reservation decision comes back, the HOF handles each case:

- **ALLOW** — the function runs normally.
- **ALLOW_WITH_CAPS** — the function runs. Caps are available through `getCyclesContext()` for the function to inspect and respect.
- **DENY** — the function does not run. A `BudgetExceededError` (or appropriate subclass) is raised.

```typescript :line-numbers
import { BudgetExceededError, CyclesProtocolError } from "runcycles";

try {
  const result = await summarize("Hello");
} catch (err) {
  if (err instanceof BudgetExceededError) { // [!code focus]
    result = fallbackResponse();
  } else if (err instanceof CyclesProtocolError) {
    if (err.retryAfterMs) {
      // retry after suggested delay
    }
    result = fallbackResponse();
  }
}
```

### Exception hierarchy

| Exception | When |
|-----------|------|
| `CyclesError` | Base for all Cycles errors |
| `CyclesProtocolError` | Server returned a protocol-level error |
| `BudgetExceededError` | Budget insufficient for the reservation |
| `OverdraftLimitExceededError` | Debt exceeds the overdraft limit |
| `DebtOutstandingError` | Outstanding debt blocks new reservations (when no overdraft limit configured) |
| `ReservationExpiredError` | Operating on an expired reservation |
| `ReservationFinalizedError` | Operating on an already-committed/released reservation |
| `CyclesTransportError` | Network-level failure (connection, DNS, timeout). Exported for user code — the SDK itself surfaces transport failures from `withCycles`/`reserveForStream` as `CyclesProtocolError` with `status: -1` |

## Streaming support

For LLM streaming where usage is only known after the stream finishes, use `reserveForStream`:

```typescript :line-numbers
import { openai } from "@ai-sdk/openai";
import { consumeStream, streamText } from "ai";
import { CyclesClient, CyclesConfig, reserveForStream } from "runcycles";

const config = new CyclesConfig({
  baseUrl: "http://localhost:7878",
  apiKey: "cyc_live_...",
  tenant: "acme",
});
const client = new CyclesClient(config);

export async function POST(request: Request) {
  const { messages } = await request.json();
  const estimate = 5000;
  const handle = await reserveForStream({ // [!code focus]
    client,
    estimate,
    actionKind: "llm.completion",
    actionName: "gpt-4o",
  });

  const settleEstimate = async (reason: string) => {
    if (handle.finalized) return;
    try {
      await handle.commit(estimate, undefined, {
        actual_source: "estimate",
        recovery_reason: reason,
      });
    } catch (settlementError) {
      console.error("Cycles settlement failed", settlementError);
    }
  };

  let result;
  try {
    result = streamText({
      model: openai("gpt-4o"),
      messages,
      abortSignal: request.signal,
      onFinish: async ({ totalUsage }) => {
        const input = totalUsage.inputTokens ?? 0;
        const output = totalUsage.outputTokens ?? 0;
        try {
          await handle.commit((input + output) * 3, { // [!code focus]
            tokensInput: input,
            tokensOutput: output,
          });
        } catch (settlementError) {
          // Known spend is already journaled before a strict commit error surfaces.
          console.error("Cycles settlement failed", settlementError);
        }
      },
      onError: async ({ error }) => {
        console.error(error);
        await settleEstimate("stream_error");
      },
      onAbort: async () => settleEstimate("stream_aborted"),
    });
  } catch (error) {
    // Synchronous setup failed before streamText dispatched the provider call.
    await handle.release("stream_startup_failed");
    throw error;
  }

  return result.toUIMessageStreamResponse({ consumeSseStream: consumeStream });
}
```

The handle is once-only and race-safe: `commit()` throws if already finalized
(so bugs are never silently hidden), while `release()` is a silent no-op if
already finalized. `onError` and `onAbort` conservatively commit the estimate
because the provider may already have produced billable partial output; release
is limited to synchronous setup failure before dispatch. Pass `consumeStream`
to preserve the AI SDK's abort callback path. In v0.4.3+, invalid actuals fall
back to the estimate with `metadata.actual_source=estimate`, and a recognized
terminal commit rejection leaves the handle finalized so broad cleanup cannot
return known-spend budget.

### Which pattern to use?

| Pattern | Use when |
|---------|----------|
| `withCycles` | You have an async function that returns a result — the lifecycle is fully automatic |
| `reserveForStream` | You're streaming and usage is known only after the stream finishes |
| `CyclesClient` | You need full control over the reservation lifecycle, or are building custom integrations |

## Programmatic client

For full control, use `CyclesClient` directly. The client operates on wire-format (snake_case) JSON. Use typed mappers for camelCase convenience, or pass raw snake_case objects:

```typescript :line-numbers
import {
  CyclesClient,
  CyclesConfig,
  reservationCreateRequestToWire,
  reservationCreateResponseFromWire,
  commitRequestToWire,
  releaseRequestToWire,
} from "runcycles";

const config = new CyclesConfig({
  baseUrl: "http://localhost:7878",
  apiKey: "cyc_live_...",
});
const client = new CyclesClient(config);

// 1. Reserve
const response = await client.createReservation( // [!code focus]
  reservationCreateRequestToWire({
    idempotencyKey: "req-001",
    subject: { tenant: "acme", agent: "support-bot" },
    action: { kind: "llm.completion", name: "gpt-4" },
    estimate: { unit: "USD_MICROCENTS", amount: 500_000 },
    ttlMs: 30_000,
  }),
);

if (!response.isSuccess) {
  throw new Error(`Reservation failed: ${response.errorMessage}`);
}

const parsed = reservationCreateResponseFromWire(response.body!);

// 2. Execute
try {
  const result = await callLlm("Hello");

  // 3. Commit
  await client.commitReservation( // [!code focus]
    parsed.reservationId!,
    commitRequestToWire({
      idempotencyKey: "commit-001",
      actual: { unit: "USD_MICROCENTS", amount: 420_000 },
      metrics: { tokensInput: 1200, tokensOutput: 800 },
    }),
  );
} catch (err) {
  // 4. Release on failure
  await client.releaseReservation(
    parsed.reservationId!,
    releaseRequestToWire({
      idempotencyKey: "release-001",
      reason: "Processing failed",
    }),
  );
  throw err;
}
```

You can also pass raw snake_case objects directly without mappers:

```typescript
const response = await client.createReservation({
  idempotency_key: "req-001",
  subject: { tenant: "acme", agent: "support-bot" },
  action: { kind: "llm.completion", name: "gpt-4" },
  estimate: { unit: "USD_MICROCENTS", amount: 500_000 },
  ttl_ms: 30_000,
});
```

### Preflight decision check

```typescript :line-numbers
import { decisionRequestToWire, decisionResponseFromWire } from "runcycles";

const response = await client.decide(
  decisionRequestToWire({
    idempotencyKey: "decide-001",
    subject: { tenant: "acme" },
    action: { kind: "llm.completion", name: "gpt-4" },
    estimate: { unit: "USD_MICROCENTS", amount: 500_000 },
  }),
);

if (response.isSuccess) {
  const parsed = decisionResponseFromWire(response.body!);
  console.log(parsed.decision); // "ALLOW", "ALLOW_WITH_CAPS", or "DENY"
}
```

### Querying balances

```typescript
import { balanceResponseFromWire } from "runcycles";

const response = await client.getBalances({ tenant: "acme" });
if (response.isSuccess) {
  const parsed = balanceResponseFromWire(response.body!);
  for (const balance of parsed.balances) {
    console.log(`${balance.scopePath}: remaining=${balance.remaining.amount}`);
  }
}
```

### Recording events (direct debit)

```typescript
import { eventCreateRequestToWire } from "runcycles";

const response = await client.createEvent( // [!code focus]
  eventCreateRequestToWire({
    idempotencyKey: "evt-001",
    subject: { tenant: "acme" },
    action: { kind: "api.call", name: "geocode" },
    actual: { unit: "USD_MICROCENTS", amount: 1_500 },
  }),
);
```

## Suggested walkthrough

Follow this order to build understanding progressively:

**1. Reserve and commit with a fixed estimate**

```typescript :line-numbers
import { CyclesClient, CyclesConfig, withCycles, setDefaultClient } from "runcycles";

const config = new CyclesConfig({
  baseUrl: "http://localhost:7878",
  apiKey: "cyc_live_...",
  tenant: "acme-corp",
});
const client = new CyclesClient(config);
setDefaultClient(client);

const hello = withCycles( // [!code focus]
  { estimate: 1000 }, // [!code focus]
  async (name: string) => `Hello, ${name}!`,
);

const result = await hello("world");
console.log(result);
```

**2. Check your balance**

```typescript
import { balanceResponseFromWire } from "runcycles";

const response = await client.getBalances({ tenant: "acme-corp" });
if (response.isSuccess) {
  console.log(balanceResponseFromWire(response.body!));
}
```

**3. Try a dry run**

```typescript
const dryRunFunc = withCycles(
  { estimate: 500, dryRun: true },
  async () => "This won't consume budget",
);

await dryRunFunc();
// Check balances — they haven't changed
```

**4. Use dynamic estimates with metrics**

```typescript :line-numbers
import { getCyclesContext } from "runcycles";

const generate = withCycles(
  {
    estimate: (prompt: string, maxTokens: number) => maxTokens * 10, // [!code focus]
    actual: (result: string) => result.length * 5, // [!code focus]
    actionKind: "llm.completion",
    actionName: "gpt-4",
  },
  async (prompt: string, maxTokens: number) => {
    const ctx = getCyclesContext();
    if (ctx) {
      ctx.metrics = { tokensInput: prompt.length, tokensOutput: maxTokens };
    }
    return `Generated response for: ${prompt}`;
  },
);

const result = await generate("Explain budgets", 500);
```

**5. Handle denials gracefully**

```typescript :line-numbers
import { BudgetExceededError } from "runcycles";

const expensiveFunc = withCycles(
  { estimate: 999_999_999 },
  async () => "This needs a lot of budget",
);

try {
  await expensiveFunc();
} catch (err) {
  if (err instanceof BudgetExceededError) { // [!code focus]
    console.log("Budget exhausted — using fallback");
  }
}
```

## Nested `withCycles` calls

Calling a `withCycles`-wrapped function from inside another `withCycles`-wrapped function is allowed — it will not throw an error. However, each wrapper creates an **independent reservation** that deducts budget separately:

```typescript
const inner = withCycles({ estimate: 100, actionName: "inner" }, async () => "done");
const outer = withCycles({ estimate: 500, actionName: "outer" }, async () => {
  return await inner(); // creates a SECOND reservation — 600 total deducted, not 500
});
```

This means nested guards **double-count budget**. The outer reservation already covers the full estimated cost of the operation, so an inner reservation deducts additional budget from the same pool.

**Recommended pattern:** Place `withCycles` at the outermost entry point only. Inner functions should be plain async functions without their own guard:

```typescript
const inner = async () => "done"; // no withCycles — called within a guarded operation

const outer = withCycles({ estimate: 500, actionName: "outer" }, async () => {
  return await inner(); // single reservation — 500 total
});
```

## Lifecycle summary

For each `withCycles`-guarded function call:

1. Estimate is evaluated (function or fixed value)
2. Reservation is created on the Cycles server
3. Decision is checked (ALLOW / ALLOW_WITH_CAPS / DENY)
4. If DENY: exception is thrown, function does not run
5. Heartbeat extension is scheduled from server-authoritative `remaining_ttl_ms` when available, with a bounded fallback for older servers
6. Function executes
7. Actual usage is evaluated (function, fixed value, or estimate)
8. Known actual usage is durably journaled, then commit is sent with the original idempotency key and optional metrics
9. Heartbeat is stopped
10. If commit remains ambiguous, the record stays queued for same-key replay; if the reservation expired, recovery switches to `POST /v1/events`
11. If function threw: reservation is released instead of committed

## Next steps

- [TypeScript Client Configuration Reference](/configuration/typescript-client-configuration-reference) — all config options and environment variables
- [Error Handling in TypeScript](/how-to/error-handling-patterns-in-typescript) — exception hierarchy, Express/Next.js patterns
- [Testing with Cycles](/how-to/testing-with-cycles) — unit and integration testing patterns
- [Using the Client Programmatically](/how-to/using-the-cycles-client-programmatically) — programmatic client reference
- [Error Handling Patterns](/how-to/error-handling-patterns-in-cycles-client-code) — general error handling patterns across all languages
- [API Reference](/api/) — interactive endpoint documentation



# How to Add Hard Budget Limits to Spring AI with Cycles

Most AI applications start with observability.

You log model usage.  
You watch provider dashboards.  
You add alerts for abnormal spend.  
You maybe enforce a few request-level limits.

That is useful.

But once your system starts running autonomous workflows, tool-calling loops, retries, or multi-step agent behavior, observability alone stops being enough.

At that point, you need a control layer that can decide **before execution** whether work is allowed to proceed.

That is where Cycles fits.

::: tip Cycles provides three runtime-authority pillars
- **Spend** — reserve-commit budget enforcement before instrumented LLM calls and tool actions
- **Risky actions** — callers can budget assigned `RISK_POINTS`; applications must apply preflight decisions and any configured caps
- **Audit** — reservations, commits, releases, and direct-usage events create lifecycle records; non-persisting preflight decisions need application logging
:::

## The problem

In a simple application, one request often maps to one model call.

In a real Spring AI system, one user action can become:

- multiple LLM calls
- retrieval steps
- tool invocations
- retries on transient failure
- multi-step planning
- background follow-up work

A provider dashboard can show this after the fact.

A rate limiter can slow it down.

Neither one guarantees that the workflow stays inside a hard budget boundary.

## What Cycles adds

Cycles adds a deterministic budget-control pattern around autonomous work:

1. **Reserve exposure before execution**
2. **Execute the model or tool call**
3. **Commit actual usage or release the remainder**

This turns budget enforcement into part of the runtime path, instead of a reporting function that happens later.

In a Spring AI application, that usually means guarding:

- model invocations
- tool-calling steps
- agent loop iterations
- workflow branches
- high-cost external actions

## The mental model

Think of Cycles as a **runtime authority for autonomous agents**.

Spring AI handles prompting, model interaction, retrieval, and orchestration.

Cycles handles:

- whether an action is allowed to proceed
- how much budget is reserved for it
- how actual usage is reconciled
- how limits apply across scopes such as tenant, workspace, app, workflow, or agent

The goal is not to replace Spring AI.

The goal is to add hard budget control to it.

## Where to integrate in a Spring AI application

There are several natural integration points.

### 1. Before a model call

Before invoking a chat model or completion model, reserve budget for the expected exposure.

This is the cleanest and most common place to start.

### 2. Before a tool invocation

If tools can create meaningful cost or side effects, reserve budget before the tool runs.

This matters for:

- external APIs
- search services
- database writes
- email dispatch
- ticket creation
- payment actions

### 3. Around an agent loop iteration

If your application runs iterative planning or autonomous loops, reserve budget per step or per iteration.

That gives you a bounded envelope around recursive behavior.

### 4. Around an entire workflow or run

You can also reserve and track at a higher scope:

- per tenant
- per workspace
- per app
- per workflow
- per agent

In practice, many systems use more than one level.

For example:

- tenant daily budget
- workflow execution budget
- model-call budget per step

## A simple integration flow

At a high level, the application flow looks like this:

### Step 1: Identify the scope

Determine which budget scopes apply.

Examples:

- tenant: `acme`
- app: `support-bot`
- workflow: `refund-assistant`

### Step 2: Estimate required exposure

Before calling the model or tool, estimate how much budget the step may need.

This does not need to be perfect.  
It just needs to be sufficient to reserve bounded room to act.

### Step 3: Reserve budget

Call Cycles to reserve budget for the step.

If reservation succeeds, continue.

If reservation fails, decide how to degrade:

- stop the action
- return a fallback response
- switch to a smaller model
- skip expensive tools
- move to a lower-cost workflow path

### Step 4: Execute the step

Run the Spring AI call, tool invocation, or workflow action.

### Step 5: Commit actual usage

Once actual usage is known, commit the real amount consumed. If the actual amount is less than the reserved estimate, the unused remainder is released automatically.

### Step 6: Release if canceled

If the work is canceled or fails before producing any usage, release the reservation explicitly to return the reserved amount to the budget pool.

## Example pattern

A simplified application flow might look like this:

1. user asks a question
2. app selects tenant and workflow scope
3. app reserves 100 units before invoking the chat model
4. Spring AI executes the model call
5. actual usage comes back as 68 units
6. app commits 68 (remaining 32 is released automatically)

If the next step wants to invoke an external tool, it goes through the same pattern again.

This is how hard budget boundaries become part of runtime execution.

## Why this works better than post-hoc limits

Many teams already have some form of usage tracking.

That is not the same as pre-execution budget control.

Post-hoc tracking tells you:

::: info
what happened after the work completed
:::

Cycles tells you:

::: info
whether the submitted estimate fits the matching budget ledgers, how much room remains, and what the instrumented work commits afterward
:::

Application authorization still decides whether the caller may perform the action; Cycles accounts for the submitted exposure.

That distinction becomes critical in long-running or multi-step systems.

Without it, you are often reacting after the expensive part has already happened.

## A common first use case

One of the best first integrations is:

**guard every Spring AI model call with a Cycles reservation**

Why start there?

Because it gives you immediate value with minimal architecture change. With [`cycles-spring-ai-starter`](https://github.com/runcycles/cycles-spring-ai-starter), this is literally a Maven dependency plus a few `cycles.spring-ai.*` properties — the `CallAdvisor` and `StreamAdvisor` auto-wire onto every `ChatClient` you build through the auto-configured `ChatClient.Builder`. No call-site changes.

You can begin by enforcing:

- per-tenant budget (set the tenant on `CyclesProperties`, or supply a `SubjectResolver` bean that pulls tenant from your authenticated principal)
- per-workflow budget
- optional per-run envelope by resolving that run ID into `subjects.workflow`

`run` is not a separate Cycles scope. The standard hierarchy is `tenant → workspace → app → workflow → agent → toolset`; a run ID stored only in `dimensions` is attribution and does not create an enforceable ledger.

Then expand to:

- tool invocations (opt in with `cyclesToolGate.wrap(myTool)`)
- retrieval steps
- external side-effecting actions

This staged rollout works well because you do not need to boil the ocean on day one.

## Shadow evaluation before enabling the advisors

The current `cycles-spring-ai-starter` advisors implement the live reserve → model call → commit/release lifecycle. They do not expose a shadow-mode property: a Cycles dry-run response deliberately has no `reservation_id`, while the advisor requires one before it invokes the model.

To evaluate policy before enabling the advisors, make an explicit Cycles reservation request with `dry_run: true` alongside the existing model path, then log the hypothetical decision and the model's actual usage in your application. Dry runs create no reservation, balance mutation, or commit; the current server does emit `reservation.denied` for denied evaluations, but not a complete record of allowed decisions and outcomes. Once those results meet your cutover criteria, enable the starter for live enforcement. See the [shadow-mode rollout guide](/how-to/shadow-mode-in-cycles-how-to-roll-out-budget-enforcement-without-breaking-production).

## Handling failure correctly

A real integration must handle more than the happy path.

That includes:

- retries
- worker crashes
- partial completion
- timeouts
- duplicate requests

This is why reserve, commit, and release are separate lifecycle events.

The application should not assume execution is always synchronous or clean.

A production integration should be designed so that:

- retries are idempotent
- duplicate actions do not double-spend
- incomplete work can be reconciled
- unused reservations do not leak forever

That is where Cycles adds real operational value beyond simple counters or provider dashboards.

## What to budget first

If you are integrating Cycles into Spring AI for the first time, start with the highest-value, easiest-to-measure boundaries.

A good initial rollout is:

- model calls
- tool invocations with external cost
- tenant-level daily or monthly budgets
- per-workflow execution envelopes

Do not start by trying to model every possible action in your system.

Start with the actions most likely to create budget surprises.

## Good first policies

Examples of useful first policies include:

- hard cap per tenant
- hard cap per workflow, with a run ID mapped to `subjects.workflow` when each execution needs its own ledger
- application-side dry-run evaluation before enabling the advisors on new workflows
- downgrade path when reservation fails
- application-configured tool fallback or denial handling
- per-workspace limits for staging vs production

These are practical controls that map well to real incidents.

## Why this matters for Spring AI teams

Spring AI makes it easier to build AI applications on the JVM.

As those applications become more autonomous, they need a way to bound total exposure, not just log it.

That is the role Cycles plays.

It brings:

- pre-execution budget checks
- retry-safe enforcement
- multi-scope budget enforcement
- live advisor enforcement after separate dry-run calibration
- a clean reserve → commit / release lifecycle

In other words, it gives Spring AI applications a way to move from “watching usage” to **governing execution**.

## Practical rollout plan

A simple rollout path looks like this:

### Phase 1: Observe
Issue explicit `dry_run: true` evaluations beside the existing model calls and retain the responses and actual usage in application telemetry.

### Phase 2: Guard core model usage
Add reservation and commit around the most expensive model calls.

### Phase 3: Expand to tools
Guard tool invocations and side-effecting actions.

### Phase 4: Add hierarchical budgets
Apply policies at tenant, app, and workflow scopes. Map a run ID to the workflow subject only when you intentionally want a separate ledger per execution.

### Phase 5: Enforce degradation paths
When reservations fail, downgrade or reroute instead of simply crashing.

That sequence keeps adoption manageable.

## Summary

If you are building with Spring AI, budget enforcement should not live only in dashboards, billing pages, or after-the-fact alerts.

It should be part of the execution path.

Cycles makes that possible by introducing a deterministic runtime pattern:

- reserve before execution
- commit actual usage afterward (unused remainder is released automatically)
- release explicitly if work is canceled
- enforce policy across scopes
- stay safe under retries and concurrency

That is how Spring AI systems move from useful prototypes to governed production runtimes.

## Next steps

For the Spring AI integration specifically:

- **Start here:** [Integrating Cycles with Spring AI](/how-to/integrating-cycles-with-spring-ai) — concrete Maven/Gradle setup, configuration reference, code examples for the auto-wired chat advisor, per-tool gating, observation convention, and the v0.3.0 extension points (`SubjectResolver` for per-request attribution, `PromptTokenEstimator` for real BPE token counts).
- **Starter on GitHub:** [`cycles-spring-ai-starter`](https://github.com/runcycles/cycles-spring-ai-starter) — Spring AI-specific advisors auto-wired onto every `ChatClient`. Companion to `cycles-spring-boot-starter` (used for non-Spring-AI Spring Boot code paths via the `@Cycles` annotation).

To explore the broader Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)



# How to Choose a First Cycles Rollout: Tenant Budgets, Run Budgets, or Model-Call Guardrails?

One of the first questions teams ask after understanding Cycles is:

**Where should we start?**

That is the right question.

A good first rollout is not the most complete rollout.  
It is the rollout that gives meaningful control quickly, with the least operational friction.

Most teams do not need to model every scope, every workflow, and every action on day one.

They need one of three practical starting points:

- **tenant budgets**
- **run budgets**
- **model-call guardrails**

Each is valid.  
Each solves a different problem.  
The best choice depends on the failure mode you are trying to prevent first.

::: warning “Run budget” is an integration pattern
Cycles has no native `run` scope. This guide uses **run budget** to mean a standard `workflow` ledger whose `subjects.workflow` value is the run ID. Every protected action in that execution must submit the same workflow subject for the shared envelope to apply. A run ID stored only in `dimensions` adds attribution, not enforcement.

The ledger bounds only instrumented actions and the exposure the application submits. The host must handle a denial by stopping or degrading the run, and must authorize tools and side effects separately.
:::

::: tip Cycles provides three runtime-authority pillars
- **Spend** — reserve-commit budget enforcement before instrumented LLM calls and tool actions
- **Risky actions** — callers can budget assigned `RISK_POINTS`; applications must apply preflight decisions and any configured caps
- **Audit** — live operations create their applicable reservation, balance, and audit records; successful reserve/commit/release/direct-debit paths do not each emit a current runtime Event, and non-persisting preflight decisions need application logging

All three rollouts on this page create the applicable live reservation, balance, and audit records when they use reservations and settlement. Dry-run and `decide` create no reservation or balance mutation; the current server emits `reservation.denied` for denied evaluations, but the application must log all responses and external outcomes for a complete record. Tenant budgets and run budgets primarily address spend; run budgets also bound risky agent loops; model-call guardrails are the lowest-friction way to start with per-call LLM spend enforcement.
:::

## The wrong way to start

A common mistake is to begin with a fully generalized policy hierarchy:

- tenant
- workspace
- app
- workflow
- agent
- toolset
- action type
- model class
- side-effect category

That may be a good end state.

It is usually a bad starting point.

Why?

Because early adoption succeeds when policy is:

- easy to explain
- easy to observe
- tied to real incidents
- narrow enough to tune
- valuable enough to justify operational change

The first Cycles rollout should solve a visible problem, not express the entire ontology of your platform.

## The three best starting points

Most first rollouts should begin with one of these:

### 1. Tenant budgets

Start here if your first priority is **platform economics and customer isolation**.

### 2. Run budgets

Start here if your first priority is **preventing runaway execution**.

### 3. Model-call guardrails

Start here if your first priority is **getting a minimal, low-friction integration into production**.

These are not competing strategies forever.  
They are different first wedges.

## Option 1: Start with tenant budgets

Tenant budgets are usually the best first rollout when the core concern is:

- one customer consuming too much
- lack of multi-tenant isolation
- weak plan enforcement
- surprise provider bills
- no hard per-account usage boundary

A tenant budget answers:

::: info
How much total exposure is this customer allowed to create?
:::

That makes it a strong economic and operational starting point.

### Why tenant budgets are attractive

Tenant budgets are easy to explain.

You can say:

- each tenant gets a daily, weekly, or monthly envelope
- all instrumented actions submitted against that ledger and unit count against the envelope
- once exhausted, certain actions stop, downgrade, or defer

This is intuitive for operators, finance, product, and customer-facing teams.

### What tenant budgets solve well

Tenant budgets are strong at:

- hard customer-level limits
- usage isolation
- paid plan enforcement
- predictable top-level exposure boundaries
- straightforward dashboarding and reporting

### What tenant budgets do not solve alone

They do not fully protect against:

- a single runaway run inside a healthy tenant
- workflow-specific over-consumption
- repeated tool loops in one execution
- noisy local failure modes that stay under tenant ceilings

So tenant budgets are often a strong commercial first step, but not always the strongest operational safety step.

### Choose tenant budgets first if:

- you are multi-tenant
- spend isolation is the immediate pain
- you need clear account-level boundaries
- customer over-consumption is more urgent than runaway loops
- you want the easiest policy story for internal stakeholders

## Option 2: Start with run budgets

Run budgets are usually the best first rollout when the core concern is:

- loops
- recursive tool use
- retry storms
- agent over-execution
- long-running workflows that drift out of bounds

A run budget answers:

::: info
How much exposure can this individual execution consume before it must stop or degrade?
:::

That makes it the strongest first step for many autonomous systems.

### Why run budgets are attractive

Run budgets map directly to the incident that usually forces teams to care:

- one agent got stuck
- one workflow kept retrying
- one process used tools too many times
- one background task consumed far more than intended

Run budgets are where “bounded execution” becomes real.

### What run budgets solve well

With consistent instrumentation and host-side denial handling, run budgets are strong at:

- denying further budgeted actions in runaway loops once the ledger lacks room
- limiting recursive tool chains
- bounding one workflow execution
- protecting against local over-consumption
- creating clear envelopes around autonomous behavior

### What run budgets do not solve alone

They do not fully protect against:

- aggregate tenant overuse across many healthy runs
- plan-level commercial limits
- one customer launching many runs in parallel
- uneven cost distribution across workflow types

So run budgets are often the strongest operational safety wedge, but not the full economic model.

### Choose run budgets first if:

- your biggest fear is runaway execution
- you have already seen loops, retries, or recursive tool behavior
- you want the fastest path to bounded autonomy
- you care more about local incident prevention than plan enforcement
- your system is agentic or workflow-heavy

## Option 3: Start with model-call guardrails

Model-call guardrails are usually the best first rollout when the core concern is:

- keeping integration simple
- proving value quickly
- getting Cycles into production with minimal architecture change
- controlling the most obvious source of cost first

A model-call guardrail means:

- reserve before a model invocation
- execute the call
- commit actual usage afterward (unused remainder is released automatically)
- or release explicitly if the call is canceled

This is often the easiest place to introduce Cycles because model calls are already clear cost events.

### Why model-call guardrails are attractive

They have the lowest integration friction.

Instead of redesigning all workflow and tool policy upfront, the team can start by guarding the most expensive or frequent model calls.

This works especially well in Spring AI or JVM systems where the first integration surface is already clear.

### What model-call guardrails solve well

They are strong at:

- putting hard checks around model spend
- proving reserve → commit in a small surface area
- enabling shadow mode quickly
- creating an adoption path without deep workflow modeling
- reducing initial rollout complexity

### What model-call guardrails do not solve alone

They do not fully protect against:

- tool costs
- side-effecting actions
- whole-run overages across many steps
- tenant-level aggregate usage
- expensive non-model paths

So they are often the easiest first rollout, but not the most complete.

### Choose model-call guardrails first if:

- you want the lowest-friction first integration
- model usage is your dominant cost center
- you are integrating through Spring AI or a similar runtime
- you want to prove Cycles value before adding broader policy
- your current system is not yet deeply autonomous but is heading there

## How to decide among the three

A simple decision rule works well.

### Choose tenant budgets when the first problem is:
**Who is allowed to consume how much overall?**

### Choose run budgets when the first problem is:
**How do we stop one execution from going too far?**

### Choose model-call guardrails when the first problem is:
**What is the smallest useful place we can start enforcing?**

That framing usually makes the right first rollout obvious.

## A quick decision matrix

### Start with tenant budgets if:
- you are multi-tenant
- customers need isolated limits
- commercial plan boundaries matter now
- leadership wants predictable account-level controls
- the platform needs a top-level spend envelope

### Start with run budgets if:
- you have agent loops or autonomous workflows
- runaway execution is the main incident class
- retries and recursive tools are common
- local execution safety matters most
- one bad run can cause meaningful damage

### Start with model-call guardrails if:
- you want a simple first integration
- you already know where model calls happen
- LLM spend is the most obvious first budget surface
- you need to demonstrate value fast
- you want to start in shadow mode with minimal disruption

## What I would recommend by system type

### Multi-tenant AI SaaS platform
Start with **tenant budgets**, then add **run budgets**.

Why:
You need commercial isolation first, but you will likely need local execution safety soon after.

### Agentic workflow platform
Start with **run budgets**, then add **tenant budgets**.

Why:
Your first operational risk is usually one execution doing too much.

### Spring AI app with growing LLM spend
Start with **model-call guardrails**, then add **run budgets**.

Why:
The fastest first integration is around model calls, but bounded execution will matter as autonomy increases.

### Internal enterprise assistant
Start with **run budgets** or **model-call guardrails**, depending on architecture.

Why:
Tenant isolation may matter less initially than preventing loops and keeping model cost bounded.

### AI gateway or proxy product
Start with **tenant budgets** plus **model-call guardrails**.

Why:
The gateway naturally sees account and request boundaries first.

## A strong default rollout path

If you want a generally strong sequence, this is a good one:

### Phase 1: Model-call guardrails
Get Cycles into the execution path around model calls.

### Phase 2: Run budgets
Add bounded execution for complete runs or workflow instances.

### Phase 3: Tenant budgets
Add top-level account or customer boundaries.

### Phase 4: Workflow-specific policies
Differentiate expensive or high-value processes.

This sequence works well because it moves from easiest integration to strongest operational safety to strongest commercial control.

## Start narrow, not broad

Another common mistake is to ask:

::: info
What is the perfect first policy model?
:::

A better question is:

::: info
What is the narrowest rollout that prevents the incident we care about most?
:::

That mindset leads to faster adoption.

The first Cycles rollout should aim for:

- one clear scope
- one clear integration point
- one clear incident class
- one clear operational win

That is enough.

## Shadow mode can reduce rollout risk

No matter which first rollout you choose, shadow mode is often the safest way to start.

That means:

- send reservation requests with `dry_run: true`
- log would-allow and would-deny responses in the application
- compare estimates with actuals from application telemetry because dry runs cannot be committed
- tune thresholds before hard enforcement

This is especially useful if you are unsure whether tenant ceilings, run envelopes, or model-level estimates are well calibrated yet.

## Common mistakes

### Mistake 1: Starting with tenant budgets when the real pain is runaway runs

This gives account-level control but may leave the main operational incident unchanged.

### Mistake 2: Starting with run budgets when the real business pressure is customer over-consumption

This improves execution safety, but may not solve the commercial problem leadership actually cares about.

### Mistake 3: Starting with every action type at once

This makes policy hard to reason about and adoption harder than necessary.

### Mistake 4: Treating model-call guardrails as the final state

They are often the best starting point, but many systems eventually need broader workflow and scope-level policy.

### Mistake 5: Choosing based on architecture purity instead of real incidents

The right first rollout is the one tied to actual pain.

## A practical recommendation

If you are unsure, use this order of preference:

- choose **run budgets** first if you already have autonomous loops or multi-step workflows
- choose **tenant budgets** first if you are multi-tenant and commercial isolation is the main concern
- choose **model-call guardrails** first if you need the lowest-friction path to proving value

That is a strong default rule.

## Summary

The best first Cycles rollout depends on what you need to control first:

- **tenant budgets** for account-level boundaries and usage isolation
- **run budgets** for bounded execution and runaway loop prevention
- **model-call guardrails** for the simplest first integration around LLM cost

Do not start by modeling everything.

Start with the scope that most directly addresses your current incident or pressure point.

That is how Cycles becomes adoptable, useful, and operationally credible.

## Next steps

To explore the Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)



Cycles provides runtime budget authority for autonomous agents. At the boundaries you instrument, it reserves spend or caller-assigned action exposure **before** an LLM call, tool, or side effect. Your application still classifies and authorizes the action and must route every protected path through the boundary. This section takes you from zero to a working integration.

## New to Cycles?

Start with **[What is Cycles?](/quickstart/what-is-cycles)** for a short orientation, then run through the **[end-to-end tutorial](/quickstart/end-to-end-tutorial)** to see the reserve-commit lifecycle in practice.

## Add Cycles to an existing application

| Stack | Guide |
|---|---|
| Python | [Python client](/quickstart/getting-started-with-the-python-client) |
| TypeScript | [TypeScript client](/quickstart/getting-started-with-the-typescript-client) |
| Rust | [Rust client](/quickstart/getting-started-with-the-rust-client) |
| Java / Spring Boot | [Spring Boot starter](/quickstart/getting-started-with-the-cycles-spring-boot-starter) |
| MCP host (Claude, Cursor, Windsurf) | [MCP server quickstart](/quickstart/getting-started-with-the-mcp-server) |

### Use an AI coding assistant to integrate

[Add Cycles with Claude or Codex](/how-to/add-cycles-with-claude-or-codex) — Claude Code, Cursor, GitHub Copilot, etc. can wire Cycles into an existing app from natural-language instructions.

## Deploy self-hosted Cycles

Cycles is Apache 2.0 and self-hosted — there is no managed cloud. Run it on your own infrastructure.

- [Architecture overview](/quickstart/architecture-overview-how-cycles-fits-together) — how the runtime server, admin API, dashboard, and events service fit together.
- [Deploy the full stack](/quickstart/deploying-the-full-cycles-stack) — server + admin + dashboard + events.
- [Self-host the runtime server](/quickstart/self-hosting-the-cycles-server) — the protocol-conforming server alone.
- [Deploy the events service](/quickstart/deploying-the-events-service) — webhook delivery and signed audit events.
- [Deploy the admin dashboard](/quickstart/deploying-the-cycles-dashboard) — UI for tenants, budgets, and audit.

## Connect AI tools via MCP

[Claude Desktop](/quickstart/mcp-claude-desktop) · [Claude Code](/quickstart/mcp-claude-code) · [Cursor](/quickstart/mcp-cursor) · [Windsurf](/quickstart/mcp-windsurf) · [MCP server over HTTP](/how-to/running-the-mcp-server-over-http)

## Plan your rollout

- [Add hard budget limits to Spring AI](/quickstart/how-to-add-hard-budget-limits-to-spring-ai-with-cycles) — the reservation pattern walked through end-to-end.
- [Choose a first rollout](/quickstart/how-to-choose-a-first-cycles-rollout-tenant-budgets-run-budgets-or-model-call-guardrails) — tenant budgets, run budgets, or model-call guardrails as a starting point.

## Next

- [**How-To Guides**](/how-to/) — recipes for integrations, budget patterns, operations, and troubleshooting.
- [**Cycles Protocol**](/protocol/) — the open specification for runtime budget authority.
- [**Why Cycles**](/why-cycles) — the case for runtime authority over autonomous systems.



# Add Cycles to Claude Code

This page is the exact setup for [Claude Code](https://claude.com/product/claude-code). For the protocol overview and reserve-commit lifecycle, see the [umbrella MCP quickstart](/quickstart/getting-started-with-the-mcp-server).

::: warning MCP availability is not enforcement
Registering this MCP server gives Claude Code access to Cycles tools — `cycles_reserve`, `cycles_commit`, `cycles_release`, and balance queries. **MCP is useful for local assistant workflows and discovery. It is not, by itself, a hard runtime control unless the host or tool harness is required to call Cycles before executing the real action.** For dispatch-path enforcement in Claude Code, [Cycles Budget Guard for Claude Code](/how-to/enforcing-budgets-in-claude-code-with-budget-guard) gates non-exempt tools with `PreToolUse` hooks. For other hosts, place the Cycles check in the execution path — SDK wrapper, gateway, or framework adapter. See [Add Cycles with Claude, Codex, Cursor, or Windsurf](/how-to/add-cycles-with-claude-or-codex) for the application-side recipe.
:::

## Prerequisites

- **Claude Code installed** ([install guide](https://code.claude.com/docs))
- **Node.js 20+ with `npx` available** — Claude Code launches the MCP server through `npx`.
- **A Cycles API key** (`cyc_live_...`) — see [API key setup](/quickstart/getting-started-with-the-mcp-server#prerequisites). Skip this for mock mode.
- **Cycles server running** locally or remote. Skip for mock mode.

## Setup

The fastest path is the CLI, passing env vars at registration so they ride with the server config:

```bash
claude mcp add \
  --transport stdio \
  --env CYCLES_API_KEY=cyc_live_... \
  --env CYCLES_BASE_URL=http://localhost:7878 \
  cycles \
  -- npx -y @runcycles/mcp-server
```

`claude mcp add` defaults to `local` scope, which records the server in your user-level Claude Code config scoped to the current project's directory. To share with teammates, add `--scope project` (or `-s project`) — it writes a committable `.mcp.json` file at the project root. For a global server available in every project, use `--scope user`.

## Project-scoped config (alternative)

If you'd rather hand-edit a committable `.mcp.json` in your project root, use `${VAR}` expansion so secrets stay out of git:

```json
{
  "mcpServers": {
    "cycles": {
      "command": "npx",
      "args": ["-y", "@runcycles/mcp-server"],
      "env": {
        "CYCLES_API_KEY": "${CYCLES_API_KEY}",
        "CYCLES_BASE_URL": "${CYCLES_BASE_URL:-http://localhost:7878}"
      }
    }
  }
}
```

Commit `.mcp.json`, but **do not commit real secrets**. Each developer sets `CYCLES_API_KEY` in their own shell or secret manager. The first time someone opens the project, Claude Code prompts to approve the new MCP server.

## Try mock mode (no API key required)

```bash
claude mcp add --transport stdio --env CYCLES_MOCK=true cycles -- npx -y @runcycles/mcp-server
```

Returns realistic synthetic responses with no Cycles backend running. Generated IDs and timestamps vary between calls; mock mode performs no live enforcement.

## Verify

In Claude Code, ask:

> Check the budget balance for tenant acme-corp

Claude Code should invoke `cycles_check_balance` (you'll see a tool-call indicator) and return the balances. List registered MCP servers any time with:

```bash
claude mcp list
```

## Common gotchas

- **Env vars not captured automatically.** `claude mcp add` records the command but does not persist your current shell env into the config (a stdio server still inherits the environment of the shell that launched `claude`; `--env` makes the values deterministic). Either pass `--env KEY=VALUE` flags at registration (recommended) or use a project `.mcp.json` with `${VAR}` expansion.
- **CLI option order matters.** `--transport`, `--env`, and `--scope` must come before the server name (`cycles`). Arguments after `--` are passed to `@runcycles/mcp-server`.
- **Native Windows needs a wrapper.** If Claude Code runs on native Windows rather than WSL, use `-- cmd /c npx -y @runcycles/mcp-server` after the server name so Windows can launch `npx`.
- **Three scopes, not two.** `local` is the default and applies only to the current project, stored in your user config. `project` writes a shared `.mcp.json` in the project root. `user` applies in every project. If the same server name appears in multiple scopes, Claude Code uses the highest-precedence definition: **local → project → user**.
- **First-time approval prompt.** Claude Code prompts before running a new MCP server. If you don't see the Cycles tools, check that you didn't miss the prompt.
- **`claude mcp add` updates Claude Code's view, not Claude Desktop's.** Each client has its own config — see [Claude Desktop setup](/quickstart/mcp-claude-desktop) if you want both.

## What Cycles adds

MCP gives Claude Code a standard way to call tools. Cycles adds runtime authority before those tools run: budget checks, risk limits, tenant scope, and reserve → commit / release accounting.

## Next steps

- [Reserve / commit lifecycle](/quickstart/getting-started-with-the-mcp-server#the-reserve-commit-lifecycle) — what the agent actually does with these tools
- [Claude Desktop setup](/quickstart/mcp-claude-desktop) — same protocol, different config
- [Cursor setup](/quickstart/mcp-cursor) · [Windsurf setup](/quickstart/mcp-windsurf)
- [HTTP transport](/how-to/running-the-mcp-server-over-http) — for shared / multi-user gateway deployments
- [Integrating Cycles with MCP](/how-to/integrating-cycles-with-mcp) — advanced patterns



# Add Cycles to Claude Desktop

This page is the exact setup for [Claude Desktop](https://claude.com/download). For the protocol overview and reserve-commit lifecycle, see the [umbrella MCP quickstart](/quickstart/getting-started-with-the-mcp-server).

::: warning MCP availability is not enforcement
Registering this MCP server gives Claude Desktop access to Cycles tools — `cycles_reserve`, `cycles_commit`, `cycles_release`, and balance queries. **MCP is useful for local assistant workflows and discovery. It is not, by itself, a hard runtime control unless the host or tool harness is required to call Cycles before executing the real action.** For production, place the Cycles check in the execution path — SDK wrapper, gateway, or framework adapter. See [Add Cycles with Claude, Codex, Cursor, or Windsurf](/how-to/add-cycles-with-claude-or-codex) for the application-side recipe.
:::

## Prerequisites

- **Claude Desktop installed** ([download](https://claude.com/download))
- **Node.js 20+** on PATH if you use the manual `npx` configuration below.
- **A Cycles API key** (`cyc_live_...`) — see [API key setup](/quickstart/getting-started-with-the-mcp-server#prerequisites). Skip this if you only want to try mock mode below.
- **Cycles server running** locally or remote. Skip this for mock mode.

## Setup

### Desktop extension (recommended)

1. Download `cycles-mcp-server-0.6.0.mcpb` from the [latest Cycles MCP Server release](https://github.com/runcycles/cycles-mcp-server/releases/latest).
2. In Claude Desktop, open **Settings → Extensions → Advanced settings → Install Extension…** and select the downloaded file.
3. Enter your Cycles server URL and API key in the extension configuration screen. To explore without a backend, enable **Mock mode** instead; mock mode is synthetic and performs no enforcement.

Claude Desktop installs the bundled server and makes the Cycles tools available without a hand-edited JSON file. Restart Claude Desktop if the tools do not appear immediately.

### Manual JSON configuration

Use this fallback when desktop extensions are disabled by policy or when you need to manage the launch command directly.

Open **Settings → Developer → Edit Config**. Or edit the file directly:

**macOS:**
```
~/Library/Application Support/Claude/claude_desktop_config.json
```

**Windows:**
```
%APPDATA%\Claude\claude_desktop_config.json
```

On macOS, paste the following, replacing `cyc_live_...` with your real API key:

```json
{
  "mcpServers": {
    "cycles": {
      "command": "npx",
      "args": ["-y", "@runcycles/mcp-server"],
      "env": {
        "CYCLES_API_KEY": "cyc_live_...",
        "CYCLES_BASE_URL": "http://localhost:7878"
      }
    }
  }
}
```

On Windows, launch the `npx.cmd` wrapper through `cmd /c`:

```json
{
  "mcpServers": {
    "cycles": {
      "command": "cmd",
      "args": ["/c", "npx", "-y", "@runcycles/mcp-server"],
      "env": {
        "CYCLES_API_KEY": "cyc_live_...",
        "CYCLES_BASE_URL": "http://localhost:7878"
      }
    }
  }
}
```

**Quit Claude Desktop completely** (cmd+Q on macOS — closing the window is not enough), then reopen. The Cycles tools should appear in the MCP indicator at the bottom of the chat.

> **Security note:** if you put `CYCLES_API_KEY` directly in this file, treat the config file as a secret. For shared machines, use a wrapper script or a local-only test key.

## Try mock mode (no API key required)

For the desktop extension, enable **Mock mode** in its configuration screen. For a manual JSON installation, drop `CYCLES_API_KEY` and `CYCLES_BASE_URL`, and set `CYCLES_MOCK` instead. The server returns realistic synthetic responses with no Cycles backend running. Generated IDs and timestamps vary between calls, and mock mode performs no live enforcement:

```json
{
  "mcpServers": {
    "cycles": {
      "command": "npx",
      "args": ["-y", "@runcycles/mcp-server"],
      "env": { "CYCLES_MOCK": "true" }
    }
  }
}
```

On Windows, keep the `command: "cmd"` and `["/c", "npx", ...]` argument prefix from the real-mode example.

Useful for trying out the tools before standing up a stack.

## Verify

In Claude Desktop, ask:

> Check the budget balance for tenant acme-corp

Claude should call `cycles_check_balance` and return the balances. If you don't see a tools indicator or the call doesn't fire, see "Common gotchas" below.

## Common gotchas

- **Indicator missing after edit.** Claude Desktop only re-reads the config on a full quit/restart. Closing the window is not enough on macOS.
- **Manual `npx` launch fails on Windows.** Make sure Node 20+ is on PATH and `where npx` resolves, then verify the config uses `command: "cmd"` with `"/c", "npx"` at the start of `args`.
- **`CYCLES_BASE_URL` reachability.** If your Cycles server is in Docker, `localhost:7878` from Claude Desktop on macOS reaches the host's localhost — that works. From inside another container, use `host.docker.internal`.
- **API key starts with `cyc_test_` not `cyc_live_`.** Test keys work but only against test budgets; if you're getting `BUDGET_NOT_FOUND` errors, double-check the tenant has a budget allocated.
- **Where are the logs?** When the indicator stays empty or tools fail silently, Claude writes MCP logs to `~/Library/Logs/Claude/` on macOS and `%APPDATA%\Claude\logs\` on Windows. Tail the `mcp*.log` files while restarting the app.

## What Cycles adds

MCP gives Claude Desktop a standard way to call tools. The Cycles server adds budget checks, caller-assigned risk budgets, tenant scope, and reserve → commit/release accounting as tools. Those tools are cooperative in Claude Desktop; hard enforcement requires a host or application boundary that Claude Desktop cannot bypass.

## Next steps

- [Reserve / commit lifecycle](/quickstart/getting-started-with-the-mcp-server#the-reserve-commit-lifecycle) — what the agent actually does with these tools
- [Claude Code setup](/quickstart/mcp-claude-code) — same protocol, different config
- [HTTP transport](/how-to/running-the-mcp-server-over-http) — for shared / multi-user gateway deployments
- [Integrating Cycles with MCP](/how-to/integrating-cycles-with-mcp) — advanced patterns: preflight decisions, graceful degradation, fire-and-forget events



# Add Cycles to Cursor

This page is the exact setup for [Cursor](https://cursor.com). For the protocol overview and reserve-commit lifecycle, see the [umbrella MCP quickstart](/quickstart/getting-started-with-the-mcp-server).

::: warning MCP availability is not enforcement
Registering this MCP server gives Cursor access to Cycles tools — `cycles_reserve`, `cycles_commit`, `cycles_release`, and balance queries. **MCP is useful for local assistant workflows and discovery. It is not, by itself, a hard runtime control unless the host or tool harness is required to call Cycles before executing the real action.** For production, place the Cycles check in the execution path — SDK wrapper, gateway, or framework adapter. See [Add Cycles with Claude, Codex, Cursor, or Windsurf](/how-to/add-cycles-with-claude-or-codex) for the application-side recipe.
:::

## Prerequisites

- **Cursor installed** ([download](https://cursor.com))
- **Node.js 20+ with `npx` available** — Cursor launches the MCP server through `npx`.
- **A Cycles API key** (`cyc_live_...`) — see [API key setup](/quickstart/getting-started-with-the-mcp-server#prerequisites). Skip for mock mode.
- **Cycles server running** locally or remote. Skip for mock mode.

## Setup

Cursor reads MCP config from two locations:

- **Project-scoped:** `.cursor/mcp.json` in the project root (commit to share with teammates)
- **User-scoped:** `~/.cursor/mcp.json` (applies to every project)

Pick one. Most teams want project-scoped so the config travels with the repo. Create the file:

```json
{
  "mcpServers": {
    "cycles": {
      "command": "npx",
      "args": ["-y", "@runcycles/mcp-server"],
      "env": {
        "CYCLES_API_KEY": "${env:CYCLES_API_KEY}",
        "CYCLES_BASE_URL": "${env:CYCLES_BASE_URL}"
      }
    }
  }
}
```

Set `CYCLES_API_KEY` and `CYCLES_BASE_URL` in the environment where Cursor can read them. Open Cursor's settings panel → MCP, and you should see `cycles` listed. Toggle it on if it isn't already. Cursor may prompt to approve the new server the first time.

## Try mock mode (no API key required)

```json
{
  "mcpServers": {
    "cycles": {
      "command": "npx",
      "args": ["-y", "@runcycles/mcp-server"],
      "env": { "CYCLES_MOCK": "true" }
    }
  }
}
```

Returns realistic synthetic responses with no Cycles backend running. Generated IDs and timestamps vary between calls; mock mode performs no live enforcement.

## Verify

In Cursor's tool-enabled Agent / Composer flow, ask:

> Check the budget balance for tenant acme-corp

Cursor should invoke `cycles_check_balance` (you'll see the tool-call expand in the chat) and return balances. The MCP indicator in Cursor's settings should show the server as connected.

## Common gotchas

- **MCP tools only fire in Cursor's agentic / tool-enabled mode**, not plain autocomplete or basic chat. The exact label varies across Cursor releases — check the mode toggle in the chat panel.
- **Project vs user scope.** If both `.cursor/mcp.json` and `~/.cursor/mcp.json` define `cycles`, project wins inside that project's workspace.
- **Use env interpolation for secrets.** Cursor expands `${env:NAME}` in MCP config fields including `env`, so project config can be shared without committing the API key. If Cursor was launched from a GUI and cannot see your shell variables, use a local `.env` via `envFile` or set the variables in your OS environment.
- **`.cursor/` should be `.gitignore`d if it contains secrets.** If you commit `.cursor/mcp.json` with the API key in plain text, it ends up in git history. Commit only env references or mock-mode config; keep any local `.env` file out of git.
- **If Cursor changes config locations**, use **Settings → MCP** to open or verify the active config file — the in-app path is the source of truth.

## What Cycles adds

MCP gives Cursor a standard way to call tools. Cycles adds runtime authority before those tools run: budget checks, risk limits, tenant scope, and reserve → commit / release accounting.

## Next steps

- [Reserve / commit lifecycle](/quickstart/getting-started-with-the-mcp-server#the-reserve-commit-lifecycle) — what the agent actually does with these tools
- [Windsurf setup](/quickstart/mcp-windsurf) — same protocol, different config
- [Claude Desktop](/quickstart/mcp-claude-desktop) · [Claude Code](/quickstart/mcp-claude-code)
- [HTTP transport](/how-to/running-the-mcp-server-over-http) — for shared / multi-user gateway deployments
- [Integrating Cycles with MCP](/how-to/integrating-cycles-with-mcp) — advanced patterns



# Add Cycles to Windsurf / Devin Desktop

[Windsurf has transitioned to Devin Desktop](https://devin.ai/desktop). The configuration paths below apply to legacy Windsurf builds; for current Devin Desktop releases, verify the active MCP configuration location in the app before applying them. For the protocol overview and reserve-commit lifecycle, see the [umbrella MCP quickstart](/quickstart/getting-started-with-the-mcp-server).

::: warning MCP availability is not enforcement
Registering this MCP server gives Windsurf access to Cycles tools — `cycles_reserve`, `cycles_commit`, `cycles_release`, and balance queries. **MCP is useful for local assistant workflows and discovery. It is not, by itself, a hard runtime control unless the host or tool harness is required to call Cycles before executing the real action.** For production, place the Cycles check in the execution path — SDK wrapper, gateway, or framework adapter. See [Add Cycles with Claude, Codex, Cursor, or Windsurf](/how-to/add-cycles-with-claude-or-codex) for the application-side recipe.
:::

## Prerequisites

- **Windsurf installed**, or a current [Devin Desktop](https://devin.ai/desktop) build with MCP support
- **Node.js 20+ with `npx` available** — Windsurf launches the MCP server through `npx`.
- **A Cycles API key** (`cyc_live_...`) — see [API key setup](/quickstart/getting-started-with-the-mcp-server#prerequisites). Skip for mock mode.
- **Cycles server running** locally or remote. Skip for mock mode.

## Setup

Edit Windsurf's MCP config file:

The fastest way to open the active config is from inside the app: **Windsurf Settings → Cascade → MCP Servers → View Raw Config**. Or open it directly:

**macOS / Linux:**
```
~/.codeium/windsurf/mcp_config.json
```

**Windows** (location varies by install — common paths):
```
%USERPROFILE%\.codeium\windsurf\mcp_config.json
%APPDATA%\Codeium\Windsurf\mcp_config.json
```

If neither path exists, use the in-app **View Raw Config** option above to find the active file.

Create the file if it doesn't exist:

```json
{
  "mcpServers": {
    "cycles": {
      "command": "npx",
      "args": ["-y", "@runcycles/mcp-server"],
      "env": {
        "CYCLES_API_KEY": "${env:CYCLES_API_KEY}",
        "CYCLES_BASE_URL": "${env:CYCLES_BASE_URL}"
      }
    }
  }
}
```

Set `CYCLES_API_KEY` and `CYCLES_BASE_URL` in the environment where Windsurf can read them. Open Windsurf's settings → Cascade → MCP servers, and `cycles` should appear in the list. Toggle on / refresh if needed.

## Try mock mode (no API key required)

```json
{
  "mcpServers": {
    "cycles": {
      "command": "npx",
      "args": ["-y", "@runcycles/mcp-server"],
      "env": { "CYCLES_MOCK": "true" }
    }
  }
}
```

Returns realistic synthetic responses with no Cycles backend running. Generated IDs and timestamps vary between calls; mock mode performs no live enforcement.

## Verify

In Windsurf's Cascade chat, ask:

> Check the budget balance for tenant acme-corp

Cascade should invoke `cycles_check_balance` and return the balances. The tool call should be visible in the chat trail.

## Common gotchas

- **Config is typically user-scoped.** Per-project / per-workspace MCP overrides are rolling out across Windsurf release channels; check Windsurf's settings panel for "MCP servers" before assuming. If your build only supports user-scoped, use a wrapper script as the `command` that reads the right secret based on the working directory to vary keys per project.
- **Cascade mode required.** MCP tools are only available in Cascade (Windsurf's agent mode), not in inline completions or plain chat.
- **Use env interpolation for secrets.** Windsurf expands `${env:NAME}` in MCP config fields including `env`, `url`, `serverUrl`, and `headers`. If Windsurf was launched from a GUI and cannot see your shell variables, set them in your OS environment or use a wrapper script.
- **Tools list refreshes on Windsurf restart.** If you edit the config and the tools don't show, fully quit and reopen Windsurf (closing the window is not enough on macOS).
- **Cascade has a total MCP tool limit** (currently 100 tools across all enabled servers). Cycles exposes only 9 tools, but if you have many MCP servers enabled at once you may hit the cap — disable unused tools in the MCP settings panel.

## What Cycles adds

MCP gives Windsurf a standard way to call tools. Cycles adds runtime authority before those tools run: budget checks, risk limits, tenant scope, and reserve → commit / release accounting.

## Next steps

- [Reserve / commit lifecycle](/quickstart/getting-started-with-the-mcp-server#the-reserve-commit-lifecycle) — what the agent actually does with these tools
- [Cursor setup](/quickstart/mcp-cursor) — same protocol, different config
- [Claude Desktop](/quickstart/mcp-claude-desktop) · [Claude Code](/quickstart/mcp-claude-code)
- [HTTP transport](/how-to/running-the-mcp-server-over-http) — for shared / multi-user gateway deployments
- [Integrating Cycles with MCP](/how-to/integrating-cycles-with-mcp) — advanced patterns



# Self-Hosting the Cycles Server

The Cycles server is a Spring Boot application that enforces budget reservations backed by Redis. This guide covers how to run it locally, with Docker, and in production.

::: tip This guide covers the runtime Cycles Server (port 7878) only
The full Cycles stack includes three services: the **Cycles Server** (runtime enforcement, covered here), the **Admin Server** (port 7979, tenant/budget management), and the optional **Events Service** (outbound webhook delivery worker; app port 7980 and management port 9980 stay internal). See [Deploying the Full Cycles Stack](/quickstart/deploying-the-full-cycles-stack) for the end-to-end guide covering all components. For a web UI on top of the stack, see [Deploy the Admin Dashboard](/quickstart/deploying-the-cycles-dashboard).
:::

## Choose your path

- **Just the runtime server** — follow this guide.
- **Full local stack (server + admin + dashboard)** — see [Deploying the Full Cycles Stack](/quickstart/deploying-the-full-cycles-stack). Recommended for evaluation.
- **Evaluate Cycles for a multi-tenant agent SaaS** — start with the [evaluation guide](/how-to/evaluate-cycles-for-agent-saas) before deciding what to deploy.
- **Not sure where Cycles fits?** [Send us your agent/tool-call flow](/contact) and we'll map where `reserve` / `commit` should sit.

## Prerequisites

- **Docker** and **Docker Compose** (for the quick path — no Java needed), or
- **Java 21+** and **Maven 3.9+** (for building from source)
- **Redis 7+** (required for Lua script compatibility)

## Quick start with Docker Compose

### Using pre-built GHCR images (recommended)

The fastest way to get the Cycles server running. No Java or Maven required:

```bash
cd cycles-server
docker compose -f docker-compose.prod.yml up -d
```

This pulls `ghcr.io/runcycles/cycles-server:0.1.25.59` (the version pinned in `docker-compose.prod.yml`) and starts it with Redis. The prod compose file requires `REDIS_PASSWORD` and `ADMIN_API_KEY` to be set and fails fast if either is missing.

::: tip Pinning versions
`docker-compose.prod.yml` pins a specific version tag so deployments are reproducible. To upgrade, update the existing pin to the new tag and re-run `docker compose -f docker-compose.prod.yml up -d`. Check [GitHub releases](https://github.com/runcycles/cycles-server/releases) for the current stable version.
:::

### Building from source with Docker

The repository includes a multi-stage Dockerfile that builds the JAR inside Docker — no local Java or Maven needed:

```bash
cd cycles-server
docker compose up -d
```

This uses `docker-compose.yml` which builds from source via the multi-stage Dockerfile.

### Full stack (with Admin Server)

To run both the Cycles Server and Admin Server together:

```bash
cd cycles-server
docker compose -f docker-compose.full-stack.yml up -d       # build from source
docker compose -f docker-compose.full-stack.prod.yml up -d   # use GHCR images
```

The full-stack compose files expect `cycles-server-admin` and `cycles-server-events` to be cloned alongside as sibling directories.

The server is available at `http://localhost:7878`.

Verify it is running:

```bash
curl http://localhost:7878/actuator/health/readiness
```

Since cycles-server 0.1.25.45 the aggregate `/actuator/health`, `/actuator/prometheus`, and the API docs/Swagger UI require an `X-Admin-API-Key` header — only the liveness/readiness probes stay public.

## Running from source

Clone the repository and build:

```bash
git clone https://github.com/runcycles/cycles-server.git
cd cycles-server/cycles-protocol-service
mvn clean package -DskipTests
```

Start Redis (if not already running):

```bash
redis-server
```

Run the server:

```bash
java -jar cycles-protocol-service-api/target/cycles-protocol-service-api-*.jar
```

The server starts on port 7878 by default.

## Configuration

The server is configured via environment variables or `application.properties`.

### Environment variables

| Variable | Default | Description |
|---|---|---|
| `REDIS_HOST` | `localhost` | Redis server hostname |
| `REDIS_PORT` | `6379` | Redis server port |
| `REDIS_PASSWORD` | (empty) | Redis password (optional) |
| `ADMIN_API_KEY` | (empty) | Admin key for the protected operational endpoints and admin-on-behalf-of dual-auth paths. Required (`:?`) in the prod compose files |
| `DASHBOARD_CORS_ORIGIN` | (empty) | Comma-separated browser origin(s) allowed for the Cycles dashboard. Passed through in the stack compose files; set the same value on the admin server |
| `WEBHOOK_SECRET_ENCRYPTION_KEY` | (empty) | Encryption key for webhook secrets, shared with `cycles-server-admin` and `cycles-server-events` |
| `CYCLES_PUBLIC_RATE_LIMIT_ENABLED` | `true` | Per-IP 429 rate limiting on the public evidence/JWKS endpoints (since 0.1.25.46) |
| `CYCLES_PUBLIC_RATE_LIMIT_REQUESTS_PER_MINUTE` | `300` | Fixed-window per-IP request budget for the public endpoints |
| `server.port` | `7878` | HTTP server port |
| `cycles.expiry.interval-ms` | `5000` | Interval for the background reservation expiry sweep (ms) |

### Application properties (excerpt)

The most commonly tuned properties from `application.properties`. See the [Server Configuration Reference](/configuration/server-configuration-reference-for-cycles) for the full list, including evidence signing, audit retention, and event emitter settings:

```properties
# Server
server.port=7878

# Redis
redis.host=${REDIS_HOST:localhost}
redis.port=${REDIS_PORT:6379}
redis.password=${REDIS_PASSWORD:}

# Admin key for protected operational endpoints + dual-auth paths
admin.api-key=${ADMIN_API_KEY:}

# Webhook secret encryption (shared key with cycles-server-admin)
webhook.secret.encryption-key=${WEBHOOK_SECRET_ENCRYPTION_KEY:}

# Reservation expiry sweep interval
cycles.expiry.interval-ms=5000

# Public evidence/JWKS endpoint rate limit (since 0.1.25.46)
cycles.public-rate-limit.enabled=${CYCLES_PUBLIC_RATE_LIMIT_ENABLED:true}
cycles.public-rate-limit.requests-per-minute=${CYCLES_PUBLIC_RATE_LIMIT_REQUESTS_PER_MINUTE:300}

# Logging
logging.level.root=INFO
logging.level.io.runcycles.protocol=INFO

# OpenAPI / Swagger UI
springdoc.api-docs.path=/api-docs
springdoc.swagger-ui.path=/swagger-ui.html
springdoc.swagger-ui.enabled=true

# Actuator (liveness/readiness probe groups enabled)
management.endpoints.web.exposure.include=health,info,prometheus
management.endpoint.health.show-details=when-authorized
management.endpoint.health.probes.enabled=true
```

## Redis connection

The server uses a JedisPool with a default maximum of 128 connections (32 max idle, 16 min idle, 2000 ms max wait). Redis 7+ is required because the Lua scripts use features not available in earlier versions.

### Redis with authentication

Set the `REDIS_PASSWORD` environment variable:

```bash
REDIS_PASSWORD=your-redis-password java -jar cycles-protocol-service-api-*.jar
```

### Redis connection pool

The default pool configuration (128 max total, 32 max idle, 16 min idle, 2000 ms max wait) is sufficient for most workloads. For high-throughput deployments, tune it with the `redis.pool.max-total`, `redis.pool.max-idle`, `redis.pool.min-idle`, and `redis.pool.max-wait-ms` properties — no code change required.

## Background expiry sweep

The server runs a background task every 5 seconds (configurable via `cycles.expiry.interval-ms`) that:

1. Scans the reservation TTL sorted set for expired entries
2. Marks expired reservations as `EXPIRED`
3. Releases their reserved budget back to the affected scopes

This ensures abandoned reservations (from crashed clients or network failures) do not permanently consume budget.

## Health checks

The server exposes Spring Boot Actuator health endpoints. The Kubernetes-style probes are public; readiness includes the Redis dependency:

```bash
# Liveness (process only)
curl http://localhost:7878/actuator/health/liveness

# Readiness (includes Redis)
curl http://localhost:7878/actuator/health/readiness

# Aggregate health — requires the admin key since 0.1.25.45
curl -H "X-Admin-API-Key: $ADMIN_API_KEY" http://localhost:7878/actuator/health
```

Since 0.1.25.45, the aggregate `/actuator/health`, `/actuator/prometheus`, `/actuator/info`, and the API docs/Swagger UI require the `X-Admin-API-Key` header. Only `/actuator/health/liveness` and `/actuator/health/readiness` remain public for orchestrators.

## Swagger UI

The server includes interactive API documentation via Swagger UI:

```
http://localhost:7878/swagger-ui.html
```

The raw OpenAPI spec is available at:

```
http://localhost:7878/api-docs
```

::: warning Admin key required; disabled in the prod compose
Since 0.1.25.45, Swagger UI and `/api-docs` require the `X-Admin-API-Key` header. The production compose files additionally disable them entirely via `SPRINGDOC_API_DOCS_ENABLED=false` and `SPRINGDOC_SWAGGER_UI_ENABLED=false`.
:::

## Production considerations

### Stateless server

The Cycles server is stateless — all state lives in Redis. You can run multiple server instances behind a load balancer without sticky sessions.

### Redis persistence

Enable Redis persistence (RDB or AOF) to survive Redis restarts without losing budget state. For production:

```
# redis.conf
appendonly yes
appendfsync everysec
```

### Redis memory

Budget data is compact. Each scope stores a few counters. Each active reservation stores its metadata. Typical memory usage is low unless you have millions of concurrent reservations.

### Security

- Always run the server behind HTTPS in production (use a reverse proxy like nginx or a cloud load balancer)
- Use strong, unique API keys per tenant
- Set `REDIS_PASSWORD` and restrict Redis network access
- Consider running Redis in a private subnet not accessible from the internet

### Scaling

For higher throughput:

- Add more Cycles server instances behind a load balancer
- Use Redis Cluster for horizontal scaling of budget state
- Tune the JedisPool connection count based on your concurrency needs

## Verifying your deployment

After starting the server, verify the full lifecycle works. You need an API key and a budget already configured via the [Cycles Admin Server](/quickstart/deploying-the-full-cycles-stack):

```bash
# Create a reservation (requires a valid API key and budget for the tenant scope)
curl -s -X POST http://localhost:7878/v1/reservations \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "idempotency_key": "test-001",
    "subject": { "tenant": "acme-corp" },
    "action": { "kind": "test", "name": "verify" },
    "estimate": { "amount": 100, "unit": "USD_MICROCENTS" }
  }'
```

If the server is configured correctly, you will receive a JSON response with a `reservation_id` and `"decision": "ALLOW"`.

If you get `BUDGET_EXCEEDED`, you need to create a budget via the admin server first. If you get `UNAUTHORIZED`, verify your API key was created correctly. See the [full stack deployment guide](/quickstart/deploying-the-full-cycles-stack) for the complete bootstrap sequence.

## Next steps

- [API Reference](/api/) — interactive endpoint documentation
- [Server Configuration Reference](/configuration/server-configuration-reference-for-cycles) — all configuration properties
- [Architecture Overview](/quickstart/architecture-overview-how-cycles-fits-together) — how the components fit together



# What is Cycles?

Cycles provides **runtime budget authority for autonomous agents**. At application-defined mandatory boundaries, it enforces spend and caller-assigned action-exposure limits **before instrumented execution**. The application remains responsible for tool authorization, argument validation, complete path coverage, and retaining non-persisting decisions and external outcomes.

::: tip Cycles provides three runtime-authority pillars
- **Spend** — reserve-commit budget enforcement before instrumented LLM calls and tool actions
- **Risky actions** — callers can budget assigned `RISK_POINTS`; applications must apply preflight decisions and any configured caps
- **Audit** — reservations, commits, releases, and direct-usage events create lifecycle records; non-persisting preflight decisions need application logging
:::

## Choose your path

- **Cap agent tool calls in JS/TS** — start with the [TypeScript client](/quickstart/getting-started-with-the-typescript-client) or the [OpenClaw budget guard](/how-to/integrating-cycles-with-openclaw).
- **Run Cycles locally to evaluate** — bring up the full stack: [runtime server, admin server, dashboard](/quickstart/deploying-the-full-cycles-stack).
- **Evaluate Cycles for a multi-tenant agent SaaS** — start with the [evaluation guide](/how-to/evaluate-cycles-for-agent-saas) for fit/no-fit framing and a 15-minute test.
- **Not sure where Cycles fits?** [Send us your agent/tool-call flow](/contact) and we'll map where `reserve` / `commit` should sit.

```python
@cycles(estimate=5000, action_kind="llm.completion", action_name="openai:gpt-4o")
def ask(prompt: str) -> str:
    return openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
    ).choices[0].message.content
# Budget is reserved before the decorated action runs. On rejection, the decorator does not call it.
```

## The problem

Autonomous systems fail differently than traditional software. A runaway agent does not just burn dollars — **it creates unbounded exposure**.

That exposure can be financial: thousands of dollars in LLM calls accumulated before anyone notices. But it can just as easily be operational: records deleted, files overwritten, emails sent, orders placed, deployments triggered. In these cases, the damage is not measured primarily in cost, but in **consequence**.

Rate limiters control velocity — requests per second. They do not control total exposure: the cumulative cost, risk, or irreversible side effects a system is allowed to create before execution is halted. Nor do they constrain what each individual action is permitted to do.

> By the time an alert fires, the system has already acted. **Observation is useful for visibility. It is not enforcement.**

## See it in action

The [Demos](/demos/) page has self-contained scenarios you can run in 60 seconds — no LLM API key required:

- **Runaway Agent Demo** — same agent, same bug, two outcomes: without Cycles the agent burns ~$10 before being force-killed. With Cycles it stops cleanly at $1.00.
- **Action Authority Demo** — a support agent handles a billing dispute in four steps. Cycles allows internal actions but blocks the customer email before it executes.

## How Cycles solves it

Cycles enforces a budget decision before the instrumented LLM calls, tool invocations, and API requests that your application routes through it. Each protected path follows the **[reserve-commit lifecycle](/glossary#reservation)**:
> Cycles enforces where you instrument it. Uninstrumented code paths are unaffected.

```
1. Reserve    →  Lock estimated amount before the action runs
2. Execute    →  Call the LLM / tool / API
3. Commit     →  Record actual usage; unused budget is released automatically
```

If an applicable budget is exhausted, a live reservation is **rejected before the protected action executes**.

::: code-group
```python [Python]
from runcycles import CyclesClient, CyclesConfig, cycles, set_default_client

client = CyclesClient(CyclesConfig.from_env())
set_default_client(client)

@cycles(estimate=5000, action_kind="llm.completion", action_name="openai:gpt-4o") # [!code focus]
def ask(prompt: str) -> str:
    return openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
    ).choices[0].message.content

# Cycles are reserved before the action, committed after, released on failure.
result = ask("Summarize this document")
```
```typescript [TypeScript]
import { CyclesClient, CyclesConfig, withCycles, setDefaultClient } from "runcycles";

const client = new CyclesClient(CyclesConfig.fromEnv());
setDefaultClient(client);

const ask = withCycles( // [!code focus]
  { estimate: 5000, actionKind: "llm.completion", actionName: "openai:gpt-4o" }, // [!code focus]
  async (prompt: string) => {
    const res = await openai.chat.completions.create({
      model: "gpt-4o",
      messages: [{ role: "user", content: prompt }],
    });
    return res.choices[0].message.content;
  },
);

const result = await ask("Summarize this document");
```
:::

## Key guarantees

| Guarantee | What it means |
|---|---|
| **Atomic reservation** | Budget is locked across all affected scopes in one operation — no partial locks |
| **Concurrency-safe** | Multiple agents sharing a budget cannot oversubscribe |
| **Idempotent** | Retries are safe; the same action cannot settle twice |
| **Pre-enforcement** | An insufficient live reservation is rejected *before* the instrumented expensive action, not after |

## Multi-level scoping

Budgets are applied hierarchically. A single reservation can enforce limits at every level simultaneously:

```
tenant → workspace → app → workflow → agent → toolset
```

For example, a reservation with `tenant=acme, workspace=prod, app=chatbot` checks budget at:
- `tenant:acme`
- `tenant:acme/workspace:prod`
- `tenant:acme/workspace:prod/app:chatbot`

Any configured ledger among those three scopes must have sufficient budget for the reservation to succeed. A derived scope with no ledger is skipped; at least one applicable ledger in the requested unit must exist.

## Architecture

<ArchDiagram />

Your application talks to the **Cycles Server** for runtime budget checks. The **Admin Server** manages tenants, API keys, and budget ledgers. The **Events Service** (optional) delivers webhook notifications asynchronously — see [Deploying the Events Service](/quickstart/deploying-the-events-service).

## Who uses Cycles

- **Platform teams** building multi-tenant agent runtimes
- **Framework authors** integrating budget enforcement into SDKs
- **Enterprise operators** needing structured cost-accountability records
- **Teams building agents** that call paid APIs autonomously

## Get started

::: tip New to Cycles? Start here.
The **[End-to-End Tutorial](/quickstart/end-to-end-tutorial)** takes you from zero to a working budget-guarded app in ~10 minutes — deploy the stack, create a tenant, mint an API key, and run your first reservation. Do this first, then pick a language client below.
:::

### Already have a running server? Pick your client

| Stack | Guide | Time |
|-------|-------|------|
| **Python** | [Python Quickstart](/quickstart/getting-started-with-the-python-client) | ~5 min |
| **TypeScript / Node.js** | [TypeScript Quickstart](/quickstart/getting-started-with-the-typescript-client) | ~5 min |
| **Spring Boot / Java** | [Spring Boot Quickstart](/quickstart/getting-started-with-the-cycles-spring-boot-starter) | ~5 min |
| **Rust** | [Rust Quickstart](/quickstart/getting-started-with-the-rust-client) | ~5 min |
| **Claude / Cursor / Windsurf** | [MCP Server Quickstart](/quickstart/getting-started-with-the-mcp-server) | ~3 min |

### Need to deploy the server?

| Scenario | Guide | Time |
|---|---|---|
| **Single-server local** | [Self-hosting the Cycles Server](/quickstart/self-hosting-the-cycles-server) | ~5 min |
| **Full stack (runtime + admin + events)** | [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack) | ~10 min |
| **Webhooks/events only** | [Deploy the Events Service](/quickstart/deploying-the-events-service) | ~5 min |
| **Admin dashboard (web UI)** | [Deploy the Admin Dashboard](/quickstart/deploying-the-cycles-dashboard) | ~10 min |

## Next steps

- **[Ballpark this for your workload](/calculators/claude-vs-gpt-cost-standalone)** — the cost calculator takes ~30 seconds and produces a shareable URL with your numbers
- [Choose a First Rollout](/quickstart/how-to-choose-a-first-cycles-rollout-tenant-budgets-run-budgets-or-model-call-guardrails) — decide your adoption strategy
- [Architecture Overview](/quickstart/architecture-overview-how-cycles-fits-together) — how the runtime, admin, and events components fit together
- [How Cycles Compares](/concepts/how-cycles-compares-to-rate-limiters-observability-provider-caps-in-app-counters-and-job-schedulers) — vs. rate limiters, observability, provider caps

## Read the foundations

For the structural arguments behind runtime authority and the cost / action / audit pillars:

- [Beyond Budget: How Cycles Controls Agent Actions, Not Just Spend](/blog/beyond-budget-how-cycles-controls-agent-actions) — why budget is one axis of three; risk and audit need their own enforcement.
- [The AI Agent Audit Trail You're Already Building](/blog/runtime-authority-byproducts-audit-trail-and-attribution-by-default) — how runtime-authority decisions become a structured audit ledger by default.
- [Runtime Authority vs Guardrails vs Observability](/blog/runtime-authority-vs-guardrails-vs-observability) — why pre-execution decisions are a different job from post-hoc tracing.
- [Agents Are Cross-Cutting. Your Controls Aren't.](/blog/agents-are-cross-cutting-your-controls-arent) — the layer argument: governance must span providers, tools, tenants, and workers.
- [Why Local-First Agent Runtimes Need Runtime Authority](/blog/every-local-first-agent-runtime-needs-budget-authority) — for teams running OpenClaw, Cline, Aider, Continue.
- [Python AI Agent Control: Cost, Risk, and Audit by Layer](/blog/python-ai-agent-control-cost-risk-audit-layers) — Python-specific six-layer view across the three pillars.


---

# Concepts



# Action Authority: Controlling What Agents Do

Budget authority controls how much an agent spends. Action authority controls what it does.

Both are dimensions of [runtime authority](/blog/what-is-runtime-authority-for-ai-agents) — the pre-execution control layer that decides whether an agent's next action should proceed. Budget authority caps financial exposure. Action authority caps operational exposure: emails sent, deploys triggered, records modified, files deleted.

## Why cost budgets are not enough

A support agent that sends 200 customer emails costs $1.40 in model tokens. A per-run budget of $100, $50, even $5 would not have stopped a single email. The risk was not monetary — it was reputational, operational, and commercial.

Dollar budgets are the wrong unit for action authority. The problem is not "the agent spent too much." The problem is "the agent did something it should not have done."

## RISK_POINTS — budgeting what money cannot measure

Cycles supports a **RISK_POINTS** unit for caller-assigned action exposure. Instead of denominating budgets in dollars or tokens, the application assigns point values to action classes based on blast radius:

| Action | Risk points | Rationale |
|--------|------------|-----------|
| Read CRM record | 0 | No side effects |
| Add internal note | 1 | Low blast radius, reversible |
| Send customer email | 50 | High blast radius, irreversible |
| Trigger deployment | 100 | Production impact |

A workflow can get a fixed risk-point budget. Every consequential action routed through the mandatory reservation boundary deducts from it. When that budget is exhausted, the boundary rejects another metered action. Application authorization still decides which tools and arguments are permitted.

## Toolset-scoped budgets

Caller-assigned exposure budgets can use **toolset-scoped budgets** — separate budgets for different categories of tools within the same agent run:

- **Internal tools** (CRM reads, note-taking) get a generous risk-point budget
- **External tools** (customer email, deploy) get a restrictive one

The agent can exhaust its email budget while an independent internal-tool budget remains available. The host decides whether to continue with read-only work. A live reservation succeeds with `ALLOW` or configured `ALLOW_WITH_CAPS`, or returns an error when budget is unavailable.

## Graceful degradation, not hard stops

Action authority does not require killing the agent. An application can select progressively stricter configured policies:

- **Normal phase**: Full tool access after application authorization
- **Restricted phase**: High-blast-radius actions disabled
- **Read-only phase**: Search and summarize only
- **Insufficient risk budget**: No further metered action in that scope

The current server does not switch these phases or tighten caps automatically as risk points are consumed. The application selects the policy or scope and enforces the returned tool-list caps.

This is the "disable" degradation strategy applied to action authority rather than cost control. See [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer).

## Next steps

- [Glossary: Action Authority](/glossary#action-authority) — formal definition
- [AI Agent Action Control: Hard Limits on Side Effects](/blog/ai-agent-action-control-hard-limits-side-effects) — deep dive on the problem and solution
- [Runaway Agent Demo](/demos/) — a budget-bound loop demonstrating the same reservation boundary; it is not an application-permission demo
- [Exposure](/concepts/exposure-why-rate-limits-leave-agents-unbounded) — the broader concept of unbounded agent risk



# Coding Agents Need Runtime Authority

Coding agents are impressive.

They can search a codebase, scaffold features, write tests, fix bugs, and compress work that used to take hours into minutes. But as autonomous execution gets faster and cheaper, the need for runtime control does not go away.

It becomes more important.

> **Model a coding agent's blast radius:** [Blast Radius Risk Calculator →](/calculators/ai-agent-blast-radius-standalone) — DROP TABLE, schema migration, deploy, and read-only rows are pre-loaded with illustrative severity factors. Set the containment slider to an assumption supported by your actual authorization and budget controls.

Coding agents and runtime authority solve different problems at different layers. A coding agent is designed to complete work. Runtime authority is designed to decide whether autonomous work is still allowed to continue, under what limits, and with what reconciliation afterward.

This article covers the **runtime-layer** problem: reservations, enforcement, retries, concurrency, and bounded execution inside a single agent run. The business-layer problem — whether the work was worth funding in the first place — is covered separately in [Why Coding Agents Do Not Replace Cycles](/concepts/why-coding-agents-do-not-replace-cycles).

## What coding agents do well

Modern coding agents excel at:

- searching and understanding large codebases
- scaffolding features from high-level descriptions
- writing and updating tests
- fixing bugs across multiple files
- refactoring code to match patterns
- generating boilerplate and configuration

These capabilities compress developer cycles. A task that required reading dozens of files, understanding dependencies, and writing careful patches can happen in a single agent run.

That speed is the point.

It is also the source of a new control problem.

## What coding agents do not control

A coding agent is optimized to finish work. It is not designed to answer:

- **How much has this run already spent?**
- **Is this tenant allowed to consume more?**
- **Should this workflow downgrade to a cheaper model?**
- **Has cumulative retry cost exceeded the budget for this task?**
- **Should execution stop because a parent scope is exhausted?**

These are not questions about code quality or correctness. They are questions about whether autonomous execution is still authorized to continue.

Most coding agents have no built-in mechanism to answer them.

## The gap between execution and authority

The distinction matters because coding agents create real cost as they run:

- **LLM inference** — every model call costs tokens
- **Tool invocations** — code search, file reads, web lookups, and API calls accumulate
- **Retries** — failed steps retry, sometimes silently, multiplying spend
- **Fan-out** — a single high-level task can expand into dozens of subtasks
- **Long-running loops** — agents that iterate on test failures or linting errors can run indefinitely

Without runtime authority, the only control is to wait until the run finishes — or until someone notices the bill.

That is observability, not enforcement.

## Why provider caps and rate limits are not enough

Provider-level spending caps and rate limits are useful safety nets, but they solve a different problem.

**Rate limits** bound how fast a system can act. They do not bound how much total exposure a system creates over time.

**Provider controls** use vendor-defined organization, project, workspace, credit, or quota scopes. Those can be valuable hard or soft boundaries, but shared provider identities do not automatically express "this tenant may spend $50 on this run" or "this agent may use 100,000 tokens for this task."

**In-app counters** are fragile under concurrency. Two agents checking the same counter simultaneously can both proceed, creating double-spend that is only visible after the fact.

Coding agents need controls that are:

- **scoped** — per tenant, per workspace, per workflow, per agent
- **pre-authorized** — checked before execution, not after
- **concurrency-safe** — correct under parallel agent execution
- **reconciled** — actual usage committed, unused budget released

## What runtime authority looks like

Runtime authority introduces a control loop around autonomous execution:

1. **Reserve** — before work begins, declare estimated cost and reserve budget against one or more scopes
2. **Execute** — proceed only if reservation succeeds
3. **Commit** — report actual usage after work completes (unused remainder is released automatically)
4. **Release** — explicitly release budget if work is canceled

This is the [reserve-commit model](/protocol/how-reserve-commit-works-in-cycles) that Cycles implements.

For coding agents, this means:

- a run can check whether budget is available before starting
- each model call or tool invocation can debit against a scoped budget
- retries can share one reservation only when its estimate covers the whole retry envelope; otherwise each retry needs another reservation against the same scoped ledger
- if the budget is exhausted, the agent receives a clear signal to stop or degrade
- operators see real-time budget consumption, not just post-hoc bills

## Scoped budgets for multi-tenant platforms

Teams running coding agents for multiple tenants or users face a harder version of this problem.

A single global cap does not help when:

- tenant A should be allowed $100/day but tenant B only $20/day
- a specific workspace within a tenant has its own limit
- one agent run should not consume more than 50% of a tenant's remaining budget

Cycles supports [hierarchical scoped budgets](/protocol/api-reference-for-the-cycles-protocol) — budgets defined at any level of a scope tree (tenant, workspace, app, workflow, agent, toolset). A single reservation checks all applicable scopes atomically.

This means a coding agent running inside a multi-tenant platform can be governed by organizational policy without any custom enforcement logic in the agent itself.

## Events for direct-debit accounting

Not all coding agent work fits the reservation pattern. Some actions have a known cost at execution time — a fixed-price API call, a per-file processing fee, a flat-rate tool invocation.

For these, Cycles supports [events](/protocol/how-events-work-in-cycles-direct-debit-without-reservation) — direct-debit operations that atomically deduct from scoped budgets without a prior reservation.

Events give coding agent platforms a way to account for every unit of work, whether or not it was estimated in advance.

## The practical takeaway

Coding agents are getting faster, cheaper, and more autonomous.

That makes runtime authority more important, not less.

Keep building capable agents. But do not assume that the agent itself is the right place to enforce spending limits, tenant isolation, or organizational policy.

Those are infrastructure concerns. They belong in a control layer that is:

- **independent** of the agent's execution logic
- **atomic** under concurrency
- **hierarchical** across organizational scopes
- **reconciled** between estimated and actual cost

That is the problem Cycles exists to solve.

## Next steps

To learn more:

- Read [Why Coding Agents Do Not Replace Cycles](/concepts/why-coding-agents-do-not-replace-cycles) for the business-layer companion to this piece
- Read [Why Rate Limits Are Not Enough](/concepts/why-rate-limits-are-not-enough-for-autonomous-systems) for the broader case for runtime authority
- Understand the [reserve-commit lifecycle](/protocol/how-reserve-commit-works-in-cycles)
- See [how events work](/protocol/how-events-work-in-cycles-direct-debit-without-reservation) for direct-debit accounting
- Explore the full [API Reference](/protocol/api-reference-for-the-cycles-protocol)
- Try the [MCP Server](/quickstart/getting-started-with-the-mcp-server) to give a coding agent direct budget tool access — zero code changes
- Get started with the [Python Client](/quickstart/getting-started-with-the-python-client) or [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- [AI Agent Budget Control: Enforce Hard Spend Limits](/blog/ai-agent-budget-control-enforce-hard-spend-limits) — how the reserve-commit pattern works at runtime



# Comparisons

Teams evaluating Cycles usually already have some controls in place. This page helps you find the right comparison for your situation.

## Quick read

| Tool | Best for | Where Cycles fits |
|---|---|---|
| LiteLLM | Provider routing, gateway reservations, agent/session limits, MCP cost tracking | Shared application ledgers and a caller-managed lifecycle across protected services |
| Helicone | Observability, caching, window cost limits | Bounds spend pre-execution instead of after the fact |
| OpenRouter | Single-API model access, per-key caps | Adds per-tenant + per-run hierarchical budgets |
| LangSmith | Tracing/evaluation; private-beta LLM Gateway spend policies | Adds reserve-commit budgets at an application boundary, including non-LLM work |
| Guardrails AI | Content validation (PII, toxicity) | Bounds spend and caller-assigned exposure, not output content |
| Rate limiter | Velocity control (req/sec) | Bounds total consumption, not just velocity |
| Provider controls | Vendor organization/project/workspace limits | Adds application scopes such as tenant and workflow; a workflow value can be keyed per run |
| DIY wrapper | Quick prototype budget logic | Production concurrency, retries, multi-tenant safety |
| **Cycles** | **Atomic scoped budget authority before protected execution** | **Complements routing, content, authorization, and observability controls** |

Need all of it in one layer? [Talk to a founder](mailto:founder@runcycles.io) about your stack, or [run the local demo](/demos/) to see enforcement in action.

## Full capability matrix

| Approach | Pre-execution? | Per-tenant? | Cost-aware? | Action control? | Degradation? | Reserve-commit? |
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| LiteLLM | Yes (reservations for supported routes) | Multiple gateway scopes | Yes, including MCP tracking | Gateway model/tool policies | Gateway routing/fallbacks | Gateway-managed reservation and reconciliation |
| Helicone | Window rate limit | Per-user/property | Yes | No | No | No |
| OpenRouter | Yes (key cap) | Per-key | Yes | No | No | No |
| LangSmith | Gateway: yes | Workspace/API key/user | Gateway: yes | No downstream tool authorization | Gateway model fallbacks | No |
| Guardrails AI | No | No | No | No | No | No |
| Rate limiter | Velocity only | Partial | No | No | No | No |
| Provider controls | Vendor-dependent soft or hard boundary | Provider identity only | Yes for covered usage | No application tool policy | Application chooses fallback | No application reserve-commit |
| DIY wrapper | Partial | Partial | Partial | No | No | No |
| **Cycles** | **Yes, when required by host** | **Yes** | **Yes** | **Caller-assigned RISK_POINTS budget; host authorizes** | **Configured caps returned; host applies** | **Yes** |

LiteLLM documents [budget reservations enabled by default](https://docs.litellm.ai/docs/proxy/users#budget-reservation), [agent iteration/session limits](https://docs.litellm.ai/docs/a2a_iteration_budgets), and [MCP cost tracking](https://docs.litellm.ai/docs/mcp_cost) (checked September 4, 2026). Reservation coverage depends on the route; session and MCP accounting should not be assumed to have identical admission semantics. Cycles' distinction is its caller-facing lifecycle and shared application scopes for instrumented operations, including work outside the gateway. See the [detailed comparison](/concepts/cycles-vs-litellm) for qualifications and an evaluation workload.

## By alternative

### Infrastructure you already run

- **[Cycles vs Rate Limiting](/concepts/cycles-vs-rate-limiting)** — rate limiters control velocity, not total consumption. An agent can stay within its request-per-second limit and still burn through an entire budget.

- **[Cycles vs Provider Cost Controls](/concepts/cycles-vs-provider-spending-caps)** — provider budgets, credits, and quotas use vendor-defined scopes and semantics. Cycles adds caller-defined budgets for instrumented application scopes.

- **[Cycles vs Custom Token Counters](/concepts/cycles-vs-custom-token-counters)** — in-app counters work until concurrency, retries, and hierarchical scopes make them unreliable.

### LLM proxies and gateways

- **[Cycles vs LiteLLM](/concepts/cycles-vs-litellm)** — LiteLLM routes, reserves gateway budgets, and provides agent/session controls and MCP accounting. Cycles applies a caller-managed lifecycle across shared application ledgers for protected model calls, paid APIs, and other operations. The host authorizes the action.

- **[Cycles vs Helicone](/concepts/cycles-vs-helicone)** — Helicone provides observability, caching, and window-based cost limits. Cycles provides cumulative budgets for caller-submitted operations; the host separately authorizes application actions.

- **[Cycles vs OpenRouter](/concepts/cycles-vs-openrouter)** — OpenRouter provides unified model access with per-key spending caps and guardrails. Cycles adds hierarchical runtime budgets and caller-assigned RISK_POINTS. OpenRouter selects the model; Cycles evaluates the budget request; the host governs the action and any delegation policy.

### Observability and content safety

- **[Cycles vs LangSmith](/concepts/cycles-vs-langsmith)** — LangSmith traces application behavior, and its private-beta LLM Gateway can enforce provider spend policies. Cycles adds application-boundary reserve-commit budgets, including instrumented non-LLM work.

- **[Cycles vs Guardrails AI](/concepts/cycles-vs-guardrails-ai)** — Guardrails AI validates content (hallucination, toxicity, PII). Cycles governs budgets and meters caller-assigned exposure; application authorization governs tools and arguments. They solve different problems and complement each other.

- **[Cycles vs LLM Proxies and Observability Tools](/blog/cycles-vs-llm-proxies-and-observability-tools)** — broader comparison of how Cycles complements the proxy and observability ecosystem.

### Build vs use

- **[You Can Vibe Code a Budget Wrapper](/blog/vibe-coding-budget-wrapper-vs-budget-authority)** — the gap between a prototype wrapper and a production runtime authority with concurrency safety, idempotency, and multi-tenant isolation.

## Full comparison

For a deep dive across all five alternative categories with capability matrices, see **[How Cycles Compares to Rate Limiters, Observability, Provider Caps, In-App Counters, and Job Schedulers](/concepts/how-cycles-compares-to-rate-limiters-observability-provider-caps-in-app-counters-and-job-schedulers)**.

## Next steps

- **[Runtime Authority vs Runtime Authorization](/concepts/runtime-authority-vs-runtime-authorization)** — how Cycles fits alongside identity-based agent governance (AWS Bedrock AgentCore Policy, Akeyless, agent IAM). Different layer, complementary not competitive.
- **[What Cycles Is Not](/concepts/what-cycles-is-not-billing-rate-limiting-orchestration-and-other-category-confusion)** — Cycles is not billing, not rate limiting, not orchestration. Clearing up category confusion.
- **[From Observability to Enforcement](/concepts/from-observability-to-enforcement-how-teams-evolve-from-dashboards-to-budget-authority)** — how teams evolve from dashboards to runtime authority.
- **[Why Rate Limits Are Not Enough](/concepts/why-rate-limits-are-not-enough-for-autonomous-systems)** — the deeper argument for why velocity controls fail for autonomous systems.



# CyclesEvidence: Verifiable Audit for Agent Decisions

Cycles enforces configured budgets for instrumented work in real time; the host retains action authorization. CyclesEvidence can make supported budget decisions **verifiable after the fact** through signed, content-addressed envelopes without querying the live ledger.

## The problem: a 200 OK is not proof

A normal Cycles response — `decision: ALLOW`, a reservation, a `409 BUDGET_EXCEEDED` — is enough for the **caller** who made the request and holds a live connection. It is useless to anyone else, later:

- An auditor reviewing what an agent was allowed to do three months ago.
- A counterparty (e.g. a payment or agent-passport system) that needs to confirm an action ran within an authorized budget, but doesn't trust the agent's self-report.
- A compliance process that must retain verifiable records for years, long after the reservation has expired from the ledger.

For all of them, "the server said ALLOW" is hearsay. CyclesEvidence replaces hearsay with a verifiable artifact.

## What it is

For each authorization lifecycle event, Cycles can emit a **CyclesEvidence envelope**: the request and response, wrapped in a JSON object that is

- **canonicalized** with [RFC 8785 JCS](https://www.rfc-editor.org/info/rfc8785/) (a deterministic byte form),
- **content-addressed** by `evidence_id` = the SHA-256 of those canonical bytes, computed with the `evidence_id` and `signature` fields both present and set to the empty string `""` — so the id *is* the integrity check, and
- **Ed25519-signed** by the Cycles server's key — so the origin is provable.

Five artifact types cover the whole lifecycle: `decide`, `reserve`, `commit`, `release`, and `error`.

Every relevant response carries an optional `cycles_evidence` reference:

```json
"cycles_evidence": {
  "evidence_id": "8403bed43e13ef7d56a8ab402a9d29ee7dd2f405e24c0cacb51068341a5e7030",
  "cycles_evidence_url": "https://cycles.example.com/v1/evidence/8403bed4…7030"
}
```

A consumer records the `evidence_id`, then fetches the signed envelope at `cycles_evidence_url` and verifies it — whenever it wants, offline. See [CyclesEvidence Envelopes](/protocol/cycles-evidence-envelopes-in-cycles) for the wire shape and verification recipe.

## Why it matters

**Tamper-evident.** Change one byte of a budget decision and the `evidence_id` no longer matches; forge the contents and the Ed25519 signature no longer verifies. You cannot quietly rewrite history. A valid signature proves the bytes came from a key, though — not that the key is the server's. That second question is *signer authority*, resolved separately against the server's published key set, which is built to keep verifying [across key rotations](/blog/rotating-keys-shouldnt-rewrite-history) (see [A Valid Signature Doesn't Tell You Who Signed It](/blog/a-valid-signature-doesnt-tell-you-who-signed-it)).

**Cross-system binding.** A receipt or agent-passport system can record a Cycles `evidence_id` and bind its own signed receipt to it — proving *"this agent's action ran within authorized budget scope X"* by composing two independent systems, without either trusting the other's live state. This is the integration that drove the feature (with [APS, the agent-passport-system](https://github.com/aeoess/agent-passport-system)).

**Denials are first-class evidence.** The highest-signal governance event is often *"the budget said no."* A non-dry `reserve` over budget surfaces as a `409 BUDGET_EXCEEDED` `error` envelope; committing an expired reservation as a `410`. Proving an action was **blocked** is frequently more valuable than proving one was allowed — and Cycles signs both.

**The full chain is reconstructable.** `decide → reserve → commit / release`, plus the error paths, each produce evidence, with the `reservation_id` carried into commit/release so the authorization → settlement chain can be rebuilt from the artifacts alone.

**Long-horizon retention.** Content-addressed, signed records are well-suited to long-horizon record-keeping — for example EU AI Act Article 12 retention — verifiable years later, independent of whether the original server is still running.

**Zero friction to produce.** The `evidence_id` is computed *synchronously* and returned in-band on the response; the expensive signing and storage happen asynchronously. Producing the proof costs the caller nothing extra — there is no separate "generate evidence" call. One consequence of the async split: a freshly returned `cycles_evidence_url` may transiently `404` while signing and storage complete — consumers should treat that as not-yet-available and retry briefly.

## What it is not

CyclesEvidence is the **receipt, not the gate.** Enforcement is the reserve-commit ledger itself; evidence does not change a real-time decision or make budgets "safer" in the moment. Its value comes after the decision: audit, dispute resolution, cross-system trust, and compliance.

It is also **off until configured.** A deployment must set a shared signing identity before any verifiable evidence is produced — see the operator [identity enablement runbook](https://github.com/runcycles/cycles-server-events/blob/main/docs/evidence-identity-enablement.md). Until then, Cycles enforces budgets exactly as before; it just doesn't emit signed receipts.

## In one line

CyclesEvidence makes a supported budget decision portable and verifiable. A complete action audit still needs correlated host authorization, tool arguments, execution results, and external outcomes.

## Related

- [CyclesEvidence Envelopes](/protocol/cycles-evidence-envelopes-in-cycles) — the envelope shape, `evidence_id` recipe, and how to verify.
- [Correlation and Tracing](/protocol/correlation-and-tracing-in-cycles) — `trace_id` ties evidence, events, and audit entries to the originating request.
- [Error Codes and Error Handling](/protocol/error-codes-and-error-handling-in-cycles) — the denial codes that surface as `error` evidence.



# Cycles vs Custom Token Counters: Build vs Buy for Agent Budget Control

Many teams that run AI agents in production begin with a token counter.

It starts the same way every time.

A developer adds a variable. After each LLM call, increment the counter by the number of tokens used. Before the next call, check if the counter has exceeded the limit.

```python
if total_tokens < max_tokens:
    response = call_llm(prompt)
    total_tokens += response.usage.total_tokens
else:
    raise BudgetExceeded()
```

This works. For a while.

It works when you have one service, one process, one agent, and one developer who understands the counter. It stops working when any of those assumptions change.

This article explains where custom token counters break, why they break, and when to replace them with a dedicated runtime authority.

> **Run the numbers for your workload:** [Cost Calculator →](/calculators/claude-vs-gpt-cost-standalone) — token counters predict; the calculator separates the prediction (rates × volume) from the enforcement layer that bounds reality.

## The natural starting point

Building your own counter is the rational first move.

The requirements seem simple:

- Track how many tokens each run uses
- Stop the run when it exceeds a limit
- Maybe track per-tenant usage for billing

A database column, a Redis key, or even an in-memory variable can handle this. The implementation takes an afternoon. It ships quickly. It solves the immediate problem.

Teams that build counters are not doing anything wrong. They are responding to a real need with the simplest possible solution.

The problems emerge later, when the system grows.

## Where custom counters break

### Concurrency: read-then-increment is a race condition

The basic counter pattern is: read the current value, check if it is under the limit, proceed, then increment.

That is a textbook time-of-check-to-time-of-use (TOCTOU) race condition.

When two agent threads run concurrently:

1. Thread A reads the counter: 900 tokens used out of 1,000 limit.
2. Thread B reads the counter: 900 tokens used out of 1,000 limit.
3. Both threads see headroom. Both proceed.
4. Thread A's call uses 200 tokens. Thread B's call uses 200 tokens.
5. Actual total: 1,300 tokens. Budget exceeded by 30%.

This is not a theoretical concern; it is a common failure mode in counters that separate the check from the update.

Solving it correctly requires atomic compare-and-swap operations, database-level locking, or serialized access. Most ad hoc counters do not implement any of these. Even when they do, the implementation is often subtly wrong — it works under light load and breaks under production concurrency.

### Multi-process and multi-service: the counter is local

A counter stored in application memory only exists in one process.

When the system scales to multiple instances, each instance has its own counter. The budget is effectively multiplied by the number of instances. Three replicas of a service, each with a 1,000-token limit, actually allow 3,000 tokens.

Moving the counter to a shared store (Redis, PostgreSQL) solves the locality problem but introduces the concurrency problem. Now every read-check-increment must be atomic across a network boundary. Latency, retries, and connection failures add complexity.

Moving to a shared store also means every protected service that makes LLM calls needs to know about the counter, use the same key scheme, and handle failures consistently. That coordination cost grows with each new service.

### No reservation model: cannot hold budget for in-flight work

Custom counters typically track what has been used. They do not track what is currently in flight.

Consider an agent that has used 800 of its 1,000 token budget. It starts a new LLM call that is estimated to use 150 tokens. While that call is in flight, another thread checks the counter and sees 800. It also starts a call. Both calls complete. The total is 1,100.

The counter was accurate at the time of the check. It just did not account for work that was already happening.

A reservation model solves this. Before the call, the system reserves 150 tokens. The counter immediately reflects 950 (800 used + 150 reserved). The next thread sees 950 and knows the budget is nearly exhausted.

After the call completes, the reservation is committed at the actual cost. If the call used only 120 tokens, the remaining 30 are released.

Building a correct reservation model on top of a simple counter is a significant engineering effort. It requires atomic reservation, commit, release, and TTL-based expiry for reservations that never complete. Most teams do not build this.

### No hierarchical scopes

A counter tracks one number against one limit.

Production systems need limits at multiple levels:

- **Tenant level:** This customer may spend $500 per month.
- **Workspace level:** This workspace may spend $100 per day.
- **Workflow level:** This workflow type may spend $10 per execution.
- **Agent level:** This agent may spend $2 per session.
- **Toolset level:** This set of tools may spend $0.50 per call.

Enforcing all of these simultaneously means a single request must check budget at multiple levels before proceeding. Each level must be decremented atomically. If any level is insufficient, the action must be denied.

Building this with ad hoc counters means maintaining separate counters per level, with correct rollup logic, atomic multi-key operations, and consistent error handling. The complexity is substantial.

Cycles supports hierarchical scopes natively. A single reservation checks all applicable scopes in one atomic operation.

### No overage policies

A custom counter has two states: under budget and over budget. The response is binary: proceed or fail.

Production systems need more nuance.

When a tenant is approaching their budget limit, the right response might not be "stop." It might be:

- Switch from GPT-4 to GPT-3.5 (cheaper, faster, good enough for this task)
- Reduce the context window from 128K tokens to 16K tokens
- Skip the optional document enrichment step
- Return a cached response instead of a live inference
- Allow the action but flag it for review

This is graceful degradation. It keeps the system running at reduced capability instead of failing hard.

Cycles supports this through its three-way decision model. When the deepest matching budget has caps configured, the system returns `ALLOW_WITH_CAPS`. The caller receives structured constraints and must apply them; the current server does not add or tighten caps automatically as the balance falls.

Implementing this on top of a custom counter requires the counter to return not just "yes" or "no" but also "how much is left" and "what constraints apply." That turns a simple counter into a policy engine. Most teams do not make that investment.

### Maintenance burden: every new service needs the same logic

When one service has a token counter, it works fine.

When five services have token counters, each implemented slightly differently, the system has five potential sources of budget accounting bugs.

Service A uses Redis with INCR. Service B uses PostgreSQL with a row lock. Service C uses an in-memory counter because "it's just a prototype." Service D was supposed to add a counter but the team ran out of time.

The result is inconsistent enforcement, duplicated logic, and fragile coordination. Every new service that makes LLM calls must re-implement the counter pattern, or integrate with whichever shared counter exists, or — most commonly — skip it and hope for the best.

Cycles centralizes runtime authority in one service. Every client integrates through the same protocol. The budget logic lives in one place. New services call the same API. There is one source of truth for budget state.

## Comparison

| | In-App Counter | Cycles |
|---|---|---|
| **Concurrency safety** | Race conditions under parallel access | Atomic reservations — no TOCTOU bugs |
| **Multi-service** | Counter is local to one process or requires custom shared store | Centralized runtime authority accessible from any service |
| **Reservation model** | None — tracks past usage, not in-flight work | Reserve before execution, commit after, release on cancel |
| **Hierarchical scopes** | Flat — one counter, one limit | Nested — tenant → workspace → app → workflow → agent → toolset |
| **Overage policies** | Binary — allow or deny | Three-way — ALLOW, ALLOW_WITH_CAPS, DENY |
| **Maintenance** | Duplicated across services, each with its own bugs | Single integration point, one protocol, one source of truth |
| **Retry handling** | Fragile — retries may double-count or skip counting | Idempotent — retries tied to the same reservation lifecycle |
| **TTL and expiry** | Manual cleanup if at all | Built-in reservation TTL with automatic expiry and release |
| **Audit trail** | Application logs, if instrumented | Structured reservation and commit records |

## The inflection point: when to move from counters to Cycles

Custom counters are not always wrong. They are a valid solution at a certain scale.

The inflection point comes when one or more of these conditions appear.

### Multiple services making LLM calls

Once budget enforcement must span more than one service, a local counter is no longer sufficient. The coordination cost of keeping multiple counters consistent exceeds the cost of adopting a centralized authority.

### Multi-tenant deployment

When different tenants share the same infrastructure and need independent budget limits, the counter must become tenant-aware. Multiplied by hierarchical scopes such as tenant, workspace, and workflow, the counter logic becomes a budget system whether you intended to build one or not.

### Production concurrency

When agents run in parallel — multiple threads, multiple instances, multiple workflows — the TOCTOU race condition becomes a real source of overspend. Solving it correctly with custom code requires careful engineering that is hard to get right and easy to break during refactoring.

### Need for graceful degradation

When the business requires more than hard cutoffs — when "switch to a cheaper model" is the right response instead of "error 403" — a binary counter is no longer expressive enough.

### Compliance or audit requirements

When the organization needs to demonstrate that every LLM call was authorized against a budget, with a clear trail of reservations and commits, ad hoc counters do not provide the necessary structure.

If none of these apply, a custom counter may be all you need. Not every system requires a dedicated runtime authority. A prototype, a single-service application with low concurrency, or an internal tool with one user can work fine with a simple counter.

But if two or more of these conditions are present, the custom counter is likely accumulating correctness debt faster than the team can repay it.

## Migration path

Moving from custom counters to Cycles does not require a big-bang migration.

### Step 1: Deploy in shadow mode

Send Cycles reservation requests with `dry_run: true` alongside your existing counter path. The server returns hypothetical decisions without creating reservations or mutating balances. Log those responses in the application so the two systems can be compared.

Compare the decisions. Does Cycles agree with your counter? Where do they diverge? A divergence may reveal a counter bug, different scope mapping, stale state, or a difference in estimate policy, so investigate it rather than assuming which system is wrong.

See [Shadow Mode Rollout](/how-to/shadow-mode-in-cycles-how-to-roll-out-budget-enforcement-without-breaking-production) for a detailed guide.

### Step 2: Validate scope configuration

Configure Cycles with the same budget limits your counters enforce. Map your counter keys to the standard Cycles scopes: tenant, workspace, app, workflow, agent, and toolset. If one counter is keyed per run, map that run ID to `subjects.workflow`; a run ID in `dimensions` is attribution only.

### Step 3: Enable enforcement on one service

Pick a lower-impact service with simple counter logic and limited concurrency. Switch its protected paths from the custom counter to live Cycles reservations, then monitor long enough to cover representative traffic and failure cases.

### Step 4: Roll out to remaining services

Move each service from its custom counter to Cycles. With each migration, the custom counter code can be removed. The budget logic converges on a single integration point.

### Step 5: Remove the custom counters

Once all services use Cycles, the custom counter code can be deleted. No more duplicated logic, no more inconsistent enforcement, no more race conditions in hand-rolled concurrency handling.

The result is a system where budget authority is centralized, concurrency-safe, and consistent across every protected service that makes LLM calls.

## The build vs buy calculation

Building a custom counter is cheap at first. The initial implementation takes hours.

Maintaining it under production conditions costs more than most teams expect:

- Debugging race conditions that only manifest under load
- Coordinating counter logic across services during refactors
- Adding hierarchical scopes after the fact
- Building reservation semantics on top of a simple increment
- Handling edge cases around retries, crashes, and partial failures
- Explaining to the team why the budget numbers do not add up

Cycles is designed to handle these concerns from the start. It is not a better counter. It is a different primitive — a runtime authority with reservation semantics, hierarchical scopes, and concurrency safety built in.

The question is not whether you can build it yourself. You can.

The question is whether budget accounting is where your team should be spending its engineering time.

## Next steps

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Try the [End-to-End Tutorial](/quickstart/end-to-end-tutorial) — zero to a working budget-guarded LLM call in ten minutes
- [How Much Do AI Agents Actually Cost?](/blog/how-much-do-ai-agents-cost) — per-token pricing across providers and why counting tokens alone isn't enough



# Cycles vs Guardrails AI: Runtime Authority vs Content Safety

Guardrails AI and Cycles both sit in the path of LLM execution.

They both add control. They both can prevent bad outcomes.

But they control different things entirely.

Guardrails AI validates **what the model says**.

Cycles controls **whether the model gets called at all**.

One is about content safety. The other is about runtime authority. They operate at different points in the execution lifecycle, solve different problems, and complement each other cleanly.

> **Run the numbers for your workload:** [Blast Radius Risk Calculator →](/calculators/ai-agent-blast-radius-standalone) — content guardrails filter input/output text; the calculator shows what *actions* the agent could still take that no content filter catches.

## What Guardrails AI does

Guardrails AI is a framework for validating LLM inputs and outputs. It wraps model calls with validators that check whether the response meets defined safety and quality criteria.

Its core capabilities include:

### Output validation

Guardrails checks whether an LLM response meets structural and content requirements. Does the output match a schema? Does it contain required fields? Is the JSON well-formed?

### Content safety rails

Guardrails can detect and filter harmful content — toxicity, bias, personally identifiable information, profanity, or any content that violates a policy. It intercepts unsafe outputs before they reach the user.

### Schema enforcement

When an application expects structured output from an LLM, Guardrails ensures the response conforms to a defined schema. If the output is malformed, Guardrails can retry the call or return a corrected version.

### Prompt injection detection

Guardrails can identify attempts to manipulate the model through adversarial inputs. It adds a layer of defense against prompt injection attacks that try to override system instructions.

### Retry and re-ask logic

When validation fails, Guardrails can automatically retry the LLM call, optionally re-asking with a corrected prompt. This creates a feedback loop that improves output quality without manual intervention.

These are valuable capabilities. Content safety and output quality are real problems that need real solutions.

But none of these capabilities address the question: should this model call happen at all, given what the system has already spent?

## What Cycles does

Cycles supplies runtime budget authority for autonomous agents through a reserve-then-commit lifecycle.

Its core capabilities include:

### Pre-execution budget enforcement

Before an instrumented model call, the host asks Cycles whether the submitted estimate fits the matching ledgers. At a mandatory boundary, a rejected reservation prevents that call from starting. Calls that bypass the integration are outside this guarantee.

### Reserve-then-commit lifecycle

Cycles reserves estimated cost before execution, then commits actual cost afterward. Commit releases any unused portion of the hold. This prevents concurrent submitted estimates from oversubscribing the same matching ledgers; actual usage above an estimate follows the commit-overage policy.

### Concurrency-safe budget tracking

When multiple agent threads or workflows run in parallel, Cycles uses atomic reservations. If $5 remains and two submitted reservations each request $4, at most one can succeed.

### Hierarchical scope enforcement

Budgets can be enforced at multiple standard subject levels simultaneously: tenant, workspace, app, workflow, agent, and toolset. A single reservation checks all applicable populated scopes in one atomic operation. To create a ledger per run, map the run ID to a standard field such as `workflow`; `run` and `action` are not native budget scopes.

### Three-way decisions

Instead of a binary allow/deny, Cycles supports three responses:

- **ALLOW** — budget is sufficient, proceed normally
- **ALLOW_WITH_CAPS** — reservation accepted with operator-configured constraints (for example, use a cheaper model or skip optional steps)
- **DENY** — budget is exhausted, do not proceed

This enables graceful degradation instead of hard failures.

## The key difference

Guardrails AI and Cycles ask fundamentally different questions.

**Guardrails asks:** Is this LLM output safe, correct, and well-formed?

**Cycles asks:** Is this LLM call authorized to execute given the remaining budget?

Guardrails operates on content. It examines what the model produced and decides whether that content should be passed through, corrected, or blocked.

Cycles operates on economics. It examines the budget state and decides whether the model should be invoked at all.

A model call can pass Guardrails validation (the output is safe and well-formed) while failing Cycles enforcement (the budget is exhausted). And vice versa — a call can be authorized by Cycles (budget is available) while being flagged by Guardrails (the output contains PII).

These are independent concerns. Neither subsumes the other.

## Comparison

| | Guardrails AI | Cycles |
|---|---|---|
| **Primary concern** | Content safety and output quality | Budget governance and cost control |
| **When it acts** | After LLM response (output validation) or before call (input validation) | Before LLM call (pre-execution budget check) |
| **What it prevents** | Toxic content, schema violations, prompt injection, PII leakage | Budget overruns, unbounded spend, cost race conditions |
| **Concurrency model** | Per-request validation (stateless) | Atomic reservations across concurrent requests (stateful) |
| **Budget awareness** | None — does not track cost or spend | Core function — reserves, commits, and tracks budget across scopes |
| **Protocol** | Python framework with validators and guards | Open protocol with reserve-commit-release lifecycle |
| **Retry behavior** | Re-asks the model with corrected prompts | Idempotent reservations — retries do not double-spend |
| **Scope** | Per-call input/output validation | Per-tenant, per-workflow, per-agent hierarchical budgets |
| **Degradation** | Can correct or filter outputs | Can downgrade model choice, reduce scope, or deny execution |

## Where Guardrails AI falls short for budget control

Guardrails AI is not designed for cost governance. That is not a criticism — it is a scope observation.

### No cumulative cost tracking

Guardrails validates each call independently. It does not maintain a running total of how much a workflow, run, or tenant has spent. It cannot answer: "Should we stop calling the model because this run has already consumed $8 of its $10 budget?"

### No pre-execution cost check

Guardrails primarily acts on the output side. It checks the response after the model has been called. By then, the cost has already been incurred. Even its input validators do not perform budget checks.

### No reservation semantics

Guardrails has no concept of reserving budget before execution and committing actual cost afterward. It cannot prevent two concurrent calls from exceeding a shared budget because it does not track budgets at all.

### No hierarchical budget scopes

Guardrails does not enforce limits at the tenant, workspace, or workflow level. It operates on individual model calls without cross-call or cross-scope awareness.

### Retries increase cost

When Guardrails re-asks the model after a validation failure, that retry costs money. There is no budget check before the retry. If the model fails validation five times, the system pays for five calls — regardless of whether the budget can absorb them.

## Where Cycles falls short for content safety

Cycles is not designed for content validation. That is equally intentional.

### No output inspection

Cycles does not examine what the model said. It does not know whether the response contains PII, toxic language, or malformed JSON. It authorized the call to happen. What the model produces is outside its scope.

### No schema enforcement

Cycles does not validate whether LLM output matches a required structure. It governs execution economics, not output structure.

### No prompt injection detection

Cycles does not inspect prompts or responses for adversarial manipulation. That is a content-layer concern, not a budget-layer concern.

### No content filtering

Cycles cannot detect or remove harmful content from model responses. It does not operate on content at all.

## Using both together

Guardrails AI and Cycles sit at different points in the execution path. They complement each other naturally.

The flow looks like this:

```
Agent decides to call an LLM
    → Cycles: Is there budget for this call?
        → DENY → Do not call the model. Return a fallback or error.
        → ALLOW_WITH_CAPS → Call a cheaper model or reduce context.
        → ALLOW → Proceed with the intended model.
    → LLM call executes
    → Guardrails: Is this output safe and well-formed?
        → FAIL → Re-ask or return corrected output.
                  (Each retry also checks Cycles for budget.)
        → PASS → Return output to the caller.
    → Cycles: Commit actual cost. Release unused reservation.
```

This creates two complementary control layers:

1. **Budget check first (Cycles).** Before spending money, verify that the budget allows it. This prevents wasted cost on calls that should never have happened.

2. **Content check second (Guardrails).** After getting a response, verify that it meets safety and quality standards. This prevents unsafe or malformed content from reaching users.

The critical detail is in the retry loop. When Guardrails triggers a re-ask, that retry should also pass through Cycles. Otherwise, repeated validation failures can create unbounded cost — the model keeps getting called, failing validation, and retrying, with no budget check on each retry.

### Example: a customer support agent

Consider an AI agent that handles customer inquiries.

**Without either tool:** The agent calls GPT-4 for every message. A confused customer sends 50 messages in a long conversation. The agent loops through tool calls, retries, and multi-step reasoning. The run costs $30. The output occasionally contains PII from the CRM lookup. Nobody catches either problem until after the fact.

**With Guardrails only:** The agent's outputs are validated for PII and toxicity. Content safety is handled. But the agent still loops through expensive calls without limit. The $30 run still happens.

**With Cycles only:** The agent's budget is capped at $5 per run. After $5, the agent degrades to a cheaper model or stops. Cost is controlled. But the outputs are not checked for PII or safety violations.

**With both:** The agent's budget is capped at $5 per run (Cycles). Each output is validated for PII and safety (Guardrails). Retries triggered by Guardrails are checked against the remaining budget (Cycles). The system is both safe and economical.

## Different problems, different layers

It is tempting to look for one tool that handles everything. That is not how production systems work.

Content safety and runtime authority are independent concerns:

- A safe output can be too expensive.
- A cheap output can be unsafe.
- A well-formed response can come from a run that already exceeded its budget.
- A budget-compliant run can produce toxic content.

Guardrails AI solves the content problem. Cycles solves the cost problem. Together, they give teams control over both what the model says and how much it costs to say it.

Neither tool is optional if you care about both.

## Next steps

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Try the [End-to-End Tutorial](/quickstart/end-to-end-tutorial) — zero to a working budget-guarded LLM call in ten minutes
- [AI Agent Budget Patterns: A Practical Guide](/blog/agent-budget-patterns-visual-guide) — six common budget patterns with trade-offs for each



# Cycles vs Helicone: Enforcement vs Observability and Rate Limiting

Helicone is a popular LLM observability and gateway platform. It logs every model call, tracks cost per request, and offers rate limiting (including cost-based limits). If you're using Helicone, you already have visibility into what your agents spend.

The question is whether visibility and rate limiting are enough — or whether you need cumulative budget enforcement and action-level control.

> **Run the numbers for your workload:** [Cost Calculator →](/calculators/claude-vs-gpt-cost-standalone) — Helicone observes spend; the calculator shows what is *in scope* before any enforcement layer fires.

## What each does

| | Helicone | Cycles |
|---|---|---|
| **Primary role** | Observability + AI gateway | Runtime authority — pre-execution enforcement |
| **Cost tracking** | Automatic, per-request, 300+ models | Per-scope cumulative budget with remaining balance |
| **Rate limiting** | Request-count and cost-per-window (via headers) | Not a rate limiter — enforces per-action budget authority |
| **Budget enforcement** | Cost-based rate limit blocks within a time window | Cumulative budget with atomic reserve-commit lifecycle |
| **Alerts** | Threshold notifications (email, Slack) | Webhook events on budget state transitions |
| **Action control** | No authorization for downstream application tools | Caller-assigned [RISK_POINTS](/glossary#risk-points) budget; host authorizes tools |
| **Multi-tenant** | Per-user/per-property rate limit segmentation | Tenant-scoped API keys with hierarchical budgets |
| **Caching** | Built-in LLM response caching | Not a caching layer |
| **Smart routing** | Cheapest-provider selection | Not a routing layer |

## Where Helicone works well

Helicone's strengths are real:

- **Cost visibility** — automatic cost calculation for 300+ models with session-level attribution
- **Cost-based rate limiting** — `Helicone-RateLimit-Policy: 500;w=3600;u=cents;s=user` caps spend per user per window
- **Caching** — deduplicates identical requests, significantly reducing costs
- **Smart routing** — selects the cheapest provider for equivalent models
- **Configurable alerts** — cost and error threshold notifications via email and Slack

For teams that need visibility and basic cost guardrails, Helicone covers the common case.

## Where the gaps appear

### 1. Window-based vs. cumulative budget

Helicone's cost-based rate limit enforces *spend per time window* (e.g., $5/hour). It does not enforce a cumulative budget state ("you have $47.23 remaining this month"). When the window resets, the limit resets — there's no carry-over, no "remaining balance" concept.

Cycles tracks cumulative budget state with a balance that decreases with each reservation and increases with each release. The budget has an `allocated`, `remaining`, `reserved`, `spent`, and `debt` balance at all times. This is the difference between a rate limit and a budget.

### 2. Rate limit headers vs. persistent budgets

Helicone rate limits are configured per-request via HTTP headers (`Helicone-RateLimit-Policy`) with low-latency enforcement and immediate 429 feedback. However, there's no persistent budget object that lives independently of the requests. If you change the header value, the limit changes. If you forget the header, there's no limit.

Cycles budgets are persistent objects created via the admin API. They exist independently of any request. The protection still depends on the host sending each governed operation through the mandatory reservation boundary.

### 3. Gateway enforcement vs. budget-state events

Helicone alerts are notifications, while its custom request- and cost-based rate limits are enforceable gateway controls that return `429` before an over-limit provider request. These are separate features.

Cycles emits events for selected registered lifecycle transitions. A live reservation without sufficient budget returns an error before protected execution; `ALLOW_WITH_CAPS` is a separately configured decision outcome, not an automatic response at a threshold.

### 4. No action-level control

Helicone controls request volume and cost. It cannot distinguish between a $0.01 search API call and a $0.01 `send_email` tool call. Both cost the same in tokens — but the email has 10,000x the blast radius.

The host can assign [RISK_POINTS](/how-to/assigning-risk-points-to-agent-tools) estimates by consequence and require a reservation before a tool attempt. For example, two 40-point email estimates consume 80 points from a 100-point budget. The host still authorizes the email and ensures every protected attempt is instrumented.

### 5. Segmented window limits, not hierarchical cumulative budgets

Helicone supports rate-limit segmentation by user, organization, custom property, or globally — meaningful isolation for many use cases. But these are window-based limits (reset per time period), not persistent hierarchical cumulative budgets with a reserve-commit ledger.

Cycles provides per-tenant isolation with hierarchical scopes (tenant → workspace → workflow → agent), where each level has its own cumulative budget with `allocated`, `remaining`, `reserved`, `spent`, and `debt` balances — derived atomically across the full scope hierarchy on every reservation.

## Better together: Helicone + Cycles

Helicone and Cycles complement each other. Running both gives you capabilities neither provides alone:

```
Request flow:
  Agent decides to act
    → Cycles: "Should this action happen?" (budget authority, RISK_POINTS)
    → Helicone: Check rate-limit policy, route, or serve cache
    → Provider: Execute (or return cached response)
    → Helicone: Log cost and trace
    → Cycles: Commit actual cost, release unused reservation
```

**What this stack gives you:**

| Capability | Who provides it |
|---|---|
| LLM response caching (deduplicate identical calls) | Helicone |
| Cheapest-provider routing | Helicone |
| Pre-execution budget authority | Cycles |
| Caller-assigned action-exposure budget | Cycles; host authorizes actions |
| Cost attribution per trace/session | Helicone |
| Cumulative budget enforcement per tenant | Cycles |
| Rate limiting per time window | Helicone |
| Per-action reserve-commit lifecycle | Cycles |
| Cost anomaly dashboard | Helicone |
| Webhook events for automated response | Cycles |

**Concrete integration scenario:** Helicone's cache deduplicates repeated requests at zero cost — this reduces the total number of actions that even reach Cycles. For uncached requests, Cycles enforces budget authority. Meanwhile, Helicone's per-session cost tracking lets you correlate Cycles' `reservation_id` with trace data for unified debugging. Helicone reduces what you spend. Cycles limits what you're allowed to spend. Together, they form both the optimization and the enforcement layer.

**Another scenario:** Helicone's cost alert fires at 80% of a soft threshold — your team sees the Slack notification. A Cycles live reservation is rejected when sufficient budget is unavailable. `ALLOW_WITH_CAPS` is a separate, operator-configured accepted outcome, not the automatic 100% response. The alert gives you time to intervene; the mandatory reservation boundary prevents the next estimate from being accepted beyond available budget.

## What Cycles does not do

Cycles is not an observability platform, a caching layer, or a router. It doesn't trace requests, deduplicate responses, or select the cheapest provider. If you need those things (and most production stacks do), you need Helicone or a comparable tool alongside Cycles. The reserve-commit lifecycle also adds [~15ms latency per action](/blog/cycles-server-performance-benchmarks) (p50) — negligible against multi-second LLM calls, but present.

## When Helicone alone is enough

- You need cost visibility and analytics more than enforcement
- Per-window rate limiting (e.g., "$5/hour per user") is sufficient
- Your agents don't have side-effecting tools (email, deploy, mutations)
- You don't need persistent cumulative budget tracking
- Single-tenant or simple multi-user segmentation

## When you need Cycles

- You need a cumulative monthly/quarterly budget with a "remaining balance"
- Your agents have tools with side effects that need action-level control
- You need multi-tenant budget isolation with hierarchical scopes
- You need atomic budget enforcement under concurrent agent load
- You need delegation attenuation for multi-agent systems

## Sources

Feature claims verified against [Helicone's custom rate-limit documentation](https://docs.helicone.ai/features/advanced-usage/custom-rate-limits) on July 24, 2026. Cycles claims are based on v0.1.25. These tools evolve quickly—check the linked docs for the latest.

## Related

- [Cycles vs LLM Proxies and Observability Tools](/blog/cycles-vs-llm-proxies-and-observability-tools) — broader comparison
- [Cycles vs LangSmith](/concepts/cycles-vs-langsmith) — similar observability comparison
- [What Is Runtime Authority](/blog/what-is-runtime-authority-for-ai-agents) — the enforcement model



# Cycles vs LangSmith: Budget Layers Compared

LangSmith is one of the most widely adopted observability platforms for LLM applications. If you're building with LangChain or LangGraph, you're probably already using it — or evaluating it.

LangSmith is no longer only an observability comparison. Its private-beta LLM Gateway can proxy supported provider calls and enforce spend policies. Cycles remains an application-boundary budget service with reserve-commit semantics. Understanding the enabled LangSmith surface and the traffic each boundary covers prevents both gaps and redundancy.

> **Run the numbers for your workload:** [Cost Calculator →](/calculators/claude-vs-gpt-cost-standalone) · [Blast Radius Risk Calculator →](/calculators/ai-agent-blast-radius-standalone) — observability records what happened; the calculators show what *will* happen at your token volume and action profile.

## What each does

| | LangSmith | Cycles |
|---|---|---|
| **When it acts** | Observability records execution; LLM Gateway evaluates policy before a proxied model call | Application reserves before protected execution and settles afterward |
| **What it answers** | "What happened?" and, in Gateway, "Does this provider request fit the spend policy?" | "Can this submitted amount be held against every matching ledger?" |
| **Core mechanism** | Tracing, evaluation, and private-beta provider proxy policies | Reserve → commit → release |
| **Cost control** | Hard Gateway caps by organization, workspace, API key, or user over hourly through monthly windows | Atomic budgets across populated tenant → workspace → app → workflow → agent → toolset scopes |
| **Application actions** | Traces tool runs; Gateway governs supported LLM-provider traffic | Can meter caller-assigned tool exposure; host still authorizes tools and arguments |
| **Settlement** | Gateway tracks spend against caps | Explicit estimate hold, best-known actual commit, and release |
| **Deployment** | LangSmith-managed Gateway is in private beta | Self-hosted service and open protocol |

## The fundamental difference

LangSmith Observability tells you what an agent did across traces and runs. That information drives optimization, debugging, evaluation, and capacity planning.

When traffic is routed through the private-beta LLM Gateway, LangSmith can also evaluate hard spend policies on each incoming provider request. Its documented scopes are organization, workspace, API key, and user, with hourly, daily, weekly, or monthly windows. A blocked request receives `402`, and the violation is attached to a trace.

Cycles operates at the boundary the application chooses. Before a protected LLM or tool call executes, the host requests an atomic hold against matching ledgers. If the reservation fails and the host honors that result, the protected operation does not run. If it succeeds, the host commits best-known actual usage or releases an unused hold.

## Where the boundaries differ

### Provider gateway versus application boundary

LangSmith's Gateway protects LLM calls routed through that proxy and currently documents OpenAI, Anthropic, Bedrock, Baseten, Fireworks, Gemini, and Vertex AI providers. Calls that bypass the Gateway, plus arbitrary application tools such as refunds, emails, database writes, and deployments, need another mandatory control point.

Cycles is provider-neutral and can sit around any instrumented operation. Coverage is not automatic: bypass paths remain outside its budget boundary, and action authorization remains application logic.

### Spend policy versus reserve-commit

LangSmith documents real-time spend tracking and a pre-execution block when a proxied request would cross a Gateway cap. Cycles exposes a different lifecycle: reserve an estimate atomically, hold it while work is in flight, then commit actual usage or release it. That distinction matters when many calls start concurrently or when an application needs to account for work that can fail after admission.

Do not infer one product's concurrency or overage semantics from the other. Validate Gateway behavior against the private-beta version you are using; configure Cycles estimation and commit-overage policy for the application boundary you own.

### Budget exposure is not tool permission

LangSmith can trace a `send_email` run. A host can separately require a Cycles `RISK_POINTS` reservation before each authorized attempt. Neither a Gateway spend policy nor a risk-point balance decides whether a recipient, tool, or argument is authorized. Keep identity, credentials, allowlists, and argument validation at the host or gateway that executes the action.

## Where Cycles stops

Cycles does not replace LangSmith. It has no:

- **Trace visualization** — no flame graphs, no chain-of-thought replay
- **Evaluation framework** — no LLM-as-judge, no dataset management
- **Prompt management** — no prompt hub, no versioning, no sharing
- **Prompt debugging** — no A/B testing, no dataset-driven evaluation
- **Latency profiling** — no per-step timing breakdown

These are observability concerns. Cycles is not an observability tool.

## How they work together

The strongest production setup uses both:

```
Application action
  → host authorization
  → Cycles reserve (application-scope budget)
  → optional LangSmith LLM Gateway (provider spend policy, redaction, credentials)
  → provider
  → Cycles commit/release + LangSmith trace
```

### Practical example

A customer support agent built with LangChain:

1. **Cycles** checks budget before each instrumented LLM call and tool invocation. If a matching budget has `max_tokens` configured, an accepted request can return `ALLOW_WITH_CAPS`; if a live reservation lacks budget, it returns an error such as `409 BUDGET_EXCEEDED`. The application applies caps or handles the denial.

2. **LangSmith Observability** traces the instrumented chain execution. If the team also uses the private-beta Gateway, provider calls pass its spend and data policies before being forwarded.

The overlap is provider spend enforcement; the differences are boundary, scope vocabulary, lifecycle, deployment, and observability depth. Cycles does not visualize a chain execution, while LangSmith's provider Gateway does not automatically govern every application tool.

### Feeding Cycles data into LangSmith

The commit metrics (`StandardMetrics` — tokens, latency, model version) attached to each commit are available through the Cycles API. Teams that want unified dashboards can:

- Tag LangSmith traces with the Cycles `reservation_id` for cross-referencing
- Use LangSmith's custom metadata to include Cycles decision outcomes (`ALLOW`, `DENY`, `ALLOW_WITH_CAPS`)
- Build alerting rules in LangSmith that flag traces where Cycles returned `ALLOW_WITH_CAPS` — indicating that configured caps applied

## Decision guide

**Use LangSmith when you need to:**
- Debug why an agent produced a bad response
- Evaluate response quality across datasets
- Profile latency across chain steps
- Track cost attribution across runs and users
- Enforce supported-provider spend policies through its private-beta LLM Gateway

**Use Cycles when you need to:**
- Reserve estimated exposure atomically before protected application work
- Use the protocol's tenant, workspace, app, workflow, agent, and toolset scope hierarchy
- Settle best-known actual usage after execution
- Meter non-LLM work and caller-assigned action exposure
- Return configured caps for the host to apply

**Use both when you need to:**
- Run agents in production with both visibility and enforcement
- Correlate rich execution traces with reserve-commit budget records
- Debug why an agent was denied (LangSmith trace + Cycles decision)
- Layer provider-gateway policies under broader application budgets

## Key points

- **Compare enabled surfaces.** LangSmith Observability is retrospective; its private-beta LLM Gateway adds pre-execution provider spend policies.
- **Compare boundaries, not slogans.** Gateway policies cover routed provider traffic. Cycles covers instrumented application paths and exposes reserve-commit settlement.
- **Authorization remains separate.** Neither cost control grants permission to invoke an application tool.
- **Use correlated records.** LangSmith traces and Cycles reservation IDs can explain both execution and budget treatment.

LangSmith behavior was rechecked on July 24, 2026 against the official [LLM Gateway overview](https://docs.langchain.com/langsmith/llm-gateway) and [spend-policy documentation](https://docs.langchain.com/langsmith/llm-gateway-spend-policies). The Gateway is documented as private beta, so verify availability and semantics for your account.

## Next steps

- [From Observability to Enforcement](/concepts/from-observability-to-enforcement-how-teams-evolve-from-dashboards-to-budget-authority) — the evolution from dashboards to runtime authority
- [Cycles vs LLM Proxies and Observability Tools](/blog/cycles-vs-llm-proxies-and-observability-tools) — how Cycles complements LiteLLM, Portkey, Helicone, and Langfuse
- [Integrating with LangChain](/how-to/integrating-cycles-with-langchain) — add Cycles to your LangChain application
- [Integrating with LangGraph](/how-to/integrating-cycles-with-langgraph) — budget governance for LangGraph workflows



# Cycles vs LiteLLM: Application and Gateway Budgets

LiteLLM routes model calls across providers with fallback, enforces gateway budgets, and documents budget reservations enabled by default. It also provides agent iteration and session-spend limits. Advance reservation alone is therefore not a Cycles distinction.

The useful comparison is the accounting boundary. Cycles exposes a caller-facing reservation lifecycle for operations the application instruments, so model calls, paid APIs, and other tool operations can consume shared application budgets. LiteLLM may already cover your workload when its gateway paths, scopes, and enforcement semantics match your requirements.

## What each does

| | LiteLLM | Cycles |
|---|---|---|
| **Primary role** | Gateway routing, fallback, budget enforcement, and cost tracking | Budget accounting for instrumented application operations |
| **Coverage** | Routed model calls; agent controls and MCP tool-cost tracking | Any operation the host protects with the lifecycle, including calls outside a gateway |
| **Budget scopes** | Keys, teams, users, customers, and other gateway dimensions; agent/session controls | Tenant → workspace → app → workflow → agent → toolset; a workflow value can identify a run |
| **Reservations** | Gateway estimates, reserves before supported provider calls, and reconciles actual cost | Caller submits an estimate, reserves before protected work, and commits actual usage |
| **Concurrency** | Reservations account for in-flight estimates; coverage and failure behavior depend on route and configuration | Atomic holds across matching provisioned ledgers; actual overages follow the configured policy |
| **Rate and iteration limits** | RPM/TPM and agent session iteration limits | Cumulative budgets; not an RPM/TPM limiter or automatic iteration counter |
| **Tool governance** | [MCP permissions](https://docs.litellm.ai/docs/mcp_control) apply to routed tools | Caller-assigned [RISK_POINTS](/glossary#risk-points) budgets; host authorizes tools and arguments |
| **Integration** | Route supported traffic through the gateway and configure controls | Instrument each protected operation and submit its scopes, estimate, and settlement |

## LiteLLM budget reservations and agent budgets

LiteLLM's [budget reservation documentation](https://docs.litellm.ai/docs/proxy/users#budget-reservation) describes this sequence:

1. Estimate request cost from the request body and model pricing.
2. Reserve capacity against the applicable budget.
3. Reject before the provider call if the reservation would exceed the budget.
4. Replace the reservation with actual cost after the response is priced.

Reservations are enabled by default. Routes without token pricing fall back to recorded-spend enforcement, and batch submission cannot reserve the full job cost. Configured budgets require a database; the separate `fail_closed_budget_enforcement` setting addresses degraded or stale budget state. Check those deployment details when evaluating a hard ceiling.

LiteLLM also documents [`max_iterations` and `max_budget_per_session`](https://docs.litellm.ai/docs/a2a_iteration_budgets). These require session attribution; the agent setup uses `require_trace_id_on_calls_by_agent`. The session-budget guide describes checking accumulated spend before calls and adding cost after successful LLM calls, with counters expiring after one hour by default. Do not assume the session limiter has the same reservation semantics as every gateway budget.

Tool costs are also in scope for LiteLLM: [MCP cost tracking](https://docs.litellm.ai/docs/mcp_cost) supports fixed tool/server prices and custom post-call cost hooks. That establishes tool accounting, not an identical pre-execution reservation guarantee for every MCP operation. Evaluate the actual route and configured controls.

## Where Cycles adds an application budget boundary

### Shared accounting across protected services

A workflow might call a model through LiteLLM, query a paid data API directly, and dispatch a metered background job. Cycles can reserve each operation against the same workflow and tenant ledgers, even when those operations use different transports or never enter the gateway.

This coverage is explicit. The host must protect every relevant dispatch, supply an estimate in a consistent unit, and settle actual usage. Cycles does not automatically discover tool calls, provider prices, or work that bypasses the integration. The [cost-estimation guide](/how-to/how-to-estimate-exposure-before-execution-practical-reservation-strategies-for-cycles) explains this responsibility.

### Application scope hierarchy

Cycles derives scopes in the order tenant → workspace → app → workflow → agent → toolset. Only submitted levels are included, and only provisioned budgets participate. A child operation can consume both a shared tenant or workflow budget and a narrower agent budget. At least one derived scope must have a budget.

LiteLLM already provides multiple budget identities and agent/session controls. The evaluation question is whether those scopes express the application's required shared ceilings. Cycles' hierarchy does not automatically discover delegation, transfer parent allocations, or enforce child permissions; the orchestrator supplies the scopes and authorizes the work.

### Explicit lifecycle and overage handling

Cycles exposes reserve, commit, release, and reservation extension to the caller. The host reserves before dispatch, commits after work, and releases a hold when work is canceled before any usage occurs. Long-running work can [extend its reservation lease](/protocol/reservation-ttl-grace-period-and-extend-in-cycles).

Atomic reservations protect estimated capacity. They do not guarantee that actual external charges stay below an estimate. Under the [authoritative protocol](/cycles-protocol-v0.yaml), `REJECT` rejects an over-estimate commit, which can leave an accounting gap for work already performed. The default `ALLOW_IF_AVAILABLE` charges the delta up to available capacity and marks an uncovered overage as over-limit, without creating debt. `ALLOW_WITH_OVERDRAFT` can record debt subject to its specified limits. Choose and test the policy explicitly; debt is not the outcome of every overage.

### Caller-assigned exposure budgets

The host can assign [RISK_POINTS to tool attempts](/how-to/assigning-risk-points-to-agent-tools) and meter their cumulative exposure separately from money. This can apply to protected operations outside the gateway. It does not establish whether a tool or its arguments are authorized; that remains application policy.

## Using LiteLLM and Cycles together

For a protected model call, both layers can enforce a budget:

```text
Application authorizes the operation
  → Cycles: reserve estimate against application scopes
  → LiteLLM: apply gateway policy, reserve supported cost, and route
  → Provider: execute if admitted
  → LiteLLM: reconcile its reservation with actual cost
  → Application: commit the operation's actual usage to Cycles
```

For a direct paid API or background job, the application uses the same Cycles lifecycle around that service's dispatch. If LiteLLM rejects before execution and no usage occurred, release the Cycles hold. If work incurred cost before failing, settle that usage; a failed response does not necessarily mean zero cost.

Each ledger serves its own budget boundary. Recording one model charge in both systems does not mean the provider charged twice; do not sum those records as separate expenses. Retries and fallbacks that incur additional charges must be included in the application's estimate and settlement.

Cycles can return configured `ALLOW_WITH_CAPS` constraints, which the host can map to a cheaper LiteLLM route. Cycles does not select the model or automatically add caps when balance becomes low.

## A workload to evaluate across several services

The following is an evaluation design with synthetic prices, not a measured product benchmark or a packaged runnable demo. Use a provisioned Cycles stack from the [end-to-end tutorial](/quickstart/end-to-end-tutorial), a configured LiteLLM gateway, and deterministic service fixtures with request logs. Record exact image versions, pricing, budget settings, and commands with any published results.

Configure a shared tenant budget of $10 and a workflow budget of $1 using `USD_MICROCENTS` (100,000,000 units per dollar). Submit the same tenant/workflow on all operations and use separate agent identifiers for concurrent workers. Begin each trial with fresh ledgers and fixed costs equal to estimates:

| Operation | Service path | Estimate and actual cost |
|---|---|---|
| Model request | Through LiteLLM to a provider fixture | $0.20 |
| Paid search | Direct HTTP service fixture | $0.30 |
| Document processing | Separate metered worker fixture | $0.40 |

Reserve all three operations before permitting any to finish. Their holds total $0.90. While they remain active, attempt another $0.20 model operation: its Cycles reservation should be rejected, with no corresponding LiteLLM or provider invocation. Commit the first three operations and verify $0.90 charged at both workflow and tenant scopes, zero remaining holds, and $0.10 workflow capacity left. The tenant charge is the same consumption aggregated at an ancestor, not another $0.90 of expense.

In separate fresh trials, cancel the search before dispatch and verify its $0.30 hold becomes available; replay an identical commit with the same idempotency key and verify no duplicate debit; then set an actual cost above its estimate and inspect the configured overage policy. A replay-safe Cycles commit does not make a service dispatch idempotent: use the service's own mechanism for that.

For the LiteLLM baseline, keep native reservations enabled, configure applicable gateway and agent/session budgets, and include supported MCP accounting when routing tools through MCP. Report exactly which operations each configuration covers. A direct service outside the gateway demonstrates a coverage boundary; it does not prove LiteLLM cannot govern a version of that operation routed through its supported interfaces.

## Choosing the boundary

LiteLLM alone may cover the budget requirement when all governed operations use supported gateway paths and its scopes, session controls, and enforcement behavior fit the workload. Concurrency, multiple tenants, or multiple agents alone do not establish a need for Cycles.

Consider adding Cycles when protected operations across several services need shared application ledgers, a caller-managed lifecycle, or separately metered exposure budgets. Account for the integration work: the host must enforce the boundary, estimate usage, handle leases and failures, and settle charges. Cycles does not provide provider routing, failover, caching, or application authorization.

## Sources

LiteLLM claims checked against its [budget documentation](https://docs.litellm.ai/docs/proxy/users#budget-reservation), [agent iteration budgets](https://docs.litellm.ai/docs/a2a_iteration_budgets), and [MCP cost tracking](https://docs.litellm.ai/docs/mcp_cost) on September 4, 2026. These are documentation claims, not results from testing a pinned LiteLLM deployment. Cycles lifecycle and scope claims follow the repository's [authoritative YAML specification](/cycles-protocol-v0.yaml); verify the deployed implementation against that contract.

## Related

- [Cycles vs LLM Proxies and Observability Tools](/blog/cycles-vs-llm-proxies-and-observability-tools) — gateway, application budget, and tracing boundaries
- [What Is Runtime Authority](/blog/what-is-runtime-authority-for-ai-agents) — the enforcement model
- [How Teams Control AI Agents Today](/blog/how-teams-control-ai-agents-today-and-where-it-breaks) — matching controls to application boundaries



# Cycles vs OpenRouter: Runtime Authority vs Routing with Guardrails

OpenRouter is an LLM routing gateway that provides unified access to many models and providers. Its current controls include workspace budgets, guardrails assignable across workspace/member/key boundaries, model and provider restrictions, zero-data-retention policies, prompt-injection filters, and sensitive-data controls.

If all protected spend flows through OpenRouter, those controls provide real preflight enforcement. The architectural question is whether gateway-level inference controls cover the same boundary as your agent workload.

> **Run the numbers for your workload:** [Cost Calculator →](/calculators/claude-vs-gpt-cost-standalone) — OpenRouter routes; the calculator shows what *cheaper-model routing* alone saves vs hard per-tenant budget enforcement.

## What each does

| | OpenRouter | Cycles |
|---|---|---|
| **Primary role** | LLM router — model selection, provider aggregation | Runtime authority — pre-execution enforcement |
| **Budget model** | Workspace spend limits for daily, weekly, monthly, and lifetime intervals; member/key guardrails | Tenant and subject scopes such as workspace, app, workflow, agent, and toolset |
| **Enforcement** | Preflight gateway check; already-dispatched requests can cause slight overage | Atomic estimate reservation before protected work, followed by actual-cost commit |
| **Coverage** | Requests routed through the OpenRouter inference gateway | Any application operation explicitly instrumented by the host |
| **Model control** | Model/provider allowlists and routing policies | Returns configured cap fields; the host maps and enforces them |
| **Action control** | No native authorization for downstream application tools | Caller-assigned [RISK_POINTS](/glossary#risk-points) budget; application authorization remains separate |
| **Multi-tenant** | Organization, workspace, member, and key controls | Tenant-scoped keys plus subject-scoped budgets |
| **Budget hierarchy** | Workspace interval budgets and inherited/assigned guardrails | Deepest matching configured subject scope |
| **Alerts** | Usage dashboard + key credit/usage introspection | Webhook events on budget state transitions (programmatic, PagerDuty/Slack) |
| **Concurrency behavior** | Preflight enforcement; in-flight inference may slightly exceed a limit | Reservation atomically consumes available budget before work starts |

## Where OpenRouter's guardrails work well

OpenRouter's guardrails system provides:

- **Workspace budgets** for daily, weekly, monthly, and lifetime spend
- **Guardrail assignment and inheritance** across workspaces, members, and keys
- **Model and provider restrictions**
- **Data controls** including zero-data-retention and sensitive-information policies
- **Prompt filters** including regex-based prompt-injection controls
- **Hard enforcement** — requests are rejected when the limit is reached

For teams routing all inference through OpenRouter, this provides meaningful cost and data-policy enforcement at the gateway.

## Where the gaps appear

### 1. Inference workspaces vs. application execution scopes

OpenRouter workspace budgets are shared across requests and can enforce daily, weekly, monthly, and lifetime limits. Member and key guardrails add finer access controls. This is meaningful governance for centrally managed inference.

The boundary is still OpenRouter inference. An agent workflow may also pay for search, browsers, sandboxes, SaaS APIs, or database operations, and may need separate limits per workflow or run even when calls share one gateway workspace. Cycles can apply budgets to those application-defined scopes and operations, provided the host instruments them.

### 2. No action-level control

OpenRouter controls inference requests, models, providers, prompts, and data policies. It does not authorize downstream application tools or their side effects.

For example, a host can assign a `RISK_POINTS` estimate to each email attempt and require a Cycles reservation before invoking the email provider. Cycles bounds the submitted cumulative exposure; the host still decides whether the email is authorized and enforces the tool call.

### 3. No graduated enforcement or programmatic alerts

OpenRouter offers dashboard-level usage alerts and per-key activity logs. But enforcement is binary: under the cap (allowed) or over the cap (rejected). There's no graduated middle ground — no "proceed but with constraints" response, no threshold-triggered webhook events for programmatic automation.

Cycles provides [three-way decisions](/glossary#three-way-decision): ALLOW, ALLOW_WITH_CAPS (proceed with constraints like model downgrade or tool restrictions), and DENY. Plus webhook events on budget state transitions (`budget.exhausted`, `budget.over_limit_entered`) that integrate with PagerDuty, Slack, and automated remediation pipelines.

### 4. Preflight spend checks vs. reserve-commit

OpenRouter checks workspace spend before routing, but requests already in flight complete. Its documentation notes that actual workspace spend can therefore slightly exceed a limit before the next request is blocked.

Cycles [reserves an estimate before the action](/blog/what-is-runtime-authority-for-ai-agents) and commits the actual amount afterward. Concurrent reservations atomically consume available capacity. Commit overages follow the configured overage policy and may be rejected, charged from remaining capacity, or recorded as debt.

### 5. Gateway-only coverage

OpenRouter can only evaluate traffic that reaches its gateway. Cycles is provider-independent and unit-independent, so the same budget protocol can cover model calls from multiple gateways alongside explicitly instrumented non-LLM operations. Cycles is not a rate limiter and does not discover uninstrumented work.

### 6. No delegation attenuation

When agent A spawns sub-agent B via an LLM call, OpenRouter sees both as independent requests from the same key. There's no way to enforce that B has a smaller budget than A, or that B can only access a subset of A's tools.

Cycles supports [authority attenuation](/blog/agent-delegation-chains-authority-attenuation-not-trust-propagation) as an application pattern: provision a narrower child agent ledger and submit both the shared ancestor and child scopes on every protected call. Cycles does not transfer balances at handoff; action masks and delegation-depth limits remain orchestration logic.

## Better together: OpenRouter + Cycles

OpenRouter and Cycles operate at different layers. Running both gives you capabilities neither provides alone:

```
Request flow:
  Agent decides to act
    → Cycles: "Should this action happen?" (budget authority, RISK_POINTS)
    → OpenRouter: check workspace/guardrail policy, then route the model call
    → Provider: Execute the call
    → OpenRouter: Track inference usage
    → Cycles: Commit actual cost, release unused reservation
```

**What this stack gives you:**

| Capability | Who provides it |
|---|---|
| Unified access to hundreds of models | OpenRouter |
| Automatic provider selection and pricing | OpenRouter |
| Pre-execution budget authority | Cycles |
| Caller-assigned action-exposure budgets | Cycles; host authorizes the action |
| Workspace interval budgets and key/member guardrails | OpenRouter |
| Hierarchical tenant/workflow/agent budgets | Cycles |
| Model and provider allowlists | OpenRouter |
| Tool allowlists and denylists | Cycles returns configured cap fields; the host enforces them |
| Credit management | OpenRouter |
| Delegation attenuation for sub-agents | Cycles (pattern via hierarchical scopes) |

**Concrete integration scenario:** OpenRouter provides access to many models through a single API. Cycles decides whether an instrumented action can reserve against the configured budget. If the deepest matching budget supplies `ALLOW_WITH_CAPS`, your application can map a returned cap to a cheaper OpenRouter model. OpenRouter handles routing; Cycles handles the budget reservation. The current Cycles server neither infers a risk profile nor adds caps automatically as the balance falls.

**Another scenario:** OpenRouter guardrails restrict a key to lower-cost models. The application authorizes email but not deploy, assigns each email a caller-defined `RISK_POINTS` amount, and requires a Cycles reservation before sending. OpenRouter enforces model access, the application enforces tool access, and Cycles bounds the submitted email exposure.

OpenRouter selects the model and provider. Cycles decides whether the configured budget can cover the submitted action estimate. The host makes and enforces the broader authorization decision. The layers are complementary, not competing.

## What Cycles does not do

Cycles is not a router or model aggregator. It doesn't provide access to hundreds of models from a single API, handle provider selection, or manage credits across providers. If you need unified multi-model access (and most teams using OpenRouter do), you need OpenRouter or a comparable tool alongside Cycles. The reserve-commit lifecycle adds [~15ms latency per action](/blog/cycles-server-performance-benchmarks) (p50) and requires cost estimation upfront — the estimate can be wrong, and overages are tracked as debt rather than prevented.

## When OpenRouter alone is enough

- All your agents do is make LLM calls (no side-effecting tools)
- Workspace budgets and assigned guardrails match the required organizational boundaries
- Slight overage from already-dispatched inference requests is acceptable
- You don't need graduated enforcement (just hard allow/deny)
- Single-team deployment without multi-tenant isolation needs

## When you need Cycles

- Agents have tools with side effects (email, deploy, database mutations)
- You need hierarchical budgets (org → team → workspace → agent)
- You need atomic budget enforcement under concurrent agent load
- You need graduated enforcement (ALLOW_WITH_CAPS for graceful degradation)
- Multi-agent delegation chains requiring authority attenuation
- Webhook events for operational alerting and automated response

## Sources

Feature claims verified against OpenRouter's [guardrails](https://openrouter.ai/docs/guides/features/guardrails) and [workspace budgets](https://openrouter.ai/docs/guides/features/workspaces/workspace-budgets) documentation on July 24, 2026. Cycles claims are based on v0.1.25. These tools evolve quickly—check the linked docs for the latest.

## Related

- [Cycles vs LLM Proxies and Observability Tools](/blog/cycles-vs-llm-proxies-and-observability-tools) — broader comparison
- [Cycles vs LiteLLM](/concepts/cycles-vs-litellm) — similar proxy comparison
- [What Is Runtime Authority](/blog/what-is-runtime-authority-for-ai-agents) — the enforcement model



# Cycles vs Provider Spending Caps: Why Platform Limits Are Not Enough

Every major LLM provider offers controls that affect cost or capacity.

Depending on the vendor and account type, those controls can include soft budget alerts, prepaid credits, model rate limits, project or workspace attribution, throughput quotas, and billing automation.

These controls are useful. They can improve visibility, constrain throughput, or stop provider access when a credit or account limit is reached.

They do not automatically create a per-run or per-tool budget model inside your application.

The relevant question is whether their scope, timing, and failure behavior match the boundary your agent application needs to enforce.

> **Run the numbers for your workload:** [Cost Calculator →](/calculators/claude-vs-gpt-cost-standalone) — use the calculator to model workload cost, then decide which controls belong at the provider, gateway, and application boundaries.

## What provider caps offer

Provider spending caps vary by vendor, but the general pattern is consistent.

### OpenAI spend alerts, hard spend limits, and billing controls

OpenAI supports monthly spend alerts and optional hard spend limits at organization and project scope. Alerts notify while traffic continues. When tracked spend reaches an applicable hard limit, affected requests return `429 insufficient_quota`; OpenAI notes that enforcement is not instantaneous, so recorded spend can slightly exceed the configured amount. Projects also support model-specific rate limits and model access controls. Prepaid billing is separate and can stop API access when credits are exhausted, although its cutoff can also be delayed.

### Anthropic credits, usage tiers, and workspaces

Anthropic bills API usage through prepaid usage credits and stops API access when those credits run out. Its Console reports cost and usage by workspace, model, and API key, while organization usage tiers impose spend and rate limits. Auto-reload settings can change whether the prepaid balance behaves like a fixed ceiling.

### Google Cloud budget alerts

Google Cloud budgets are notification-oriented and do not automatically cap Vertex AI spend. Quotas constrain capacity rather than dollars; for newer generative models, Dynamic Shared Quota has no customer-configured predefined usage limit. Provisioned Throughput provides a separate fixed-capacity purchasing model.

### AWS Bedrock service quotas

AWS provides Bedrock service quotas that constrain request or token throughput. AWS Budgets is a separate billing service with alerts and configurable actions; those controls do not inherently represent an individual agent run or application tenant.

### The common thread

The exact behavior differs by product and plan, but provider-native controls generally share several boundaries:

- They govern traffic or billing inside one provider.
- Their identities are provider projects, workspaces, accounts, keys, or cloud projects—not necessarily your application's tenant and run hierarchy.
- Budget reporting and hard-stop semantics vary; a field labeled “budget” may be an alert threshold rather than a cap.
- Rate and throughput quotas bound request volume or capacity, not arbitrary application-side side effects.

They remain valuable controls. The gap appears when the application needs a cumulative budget for a business-defined scope or for work that is not itself a provider request.

The problem starts when teams need more than basic protection. The single-provider point in particular has its own structural argument — see [Agents Are Cross-Cutting. Your Controls Aren't.](/blog/agents-are-cross-cutting-your-controls-arent) for why a control that lives inside one provider can't reach across an agent that spans many.

## Why provider caps are not sufficient

### Provider periods are not application runs

Many provider cost controls use calendar windows, credit balances, usage tiers, or throughput intervals.

But autonomous agents operate in runs. A single agent run might take 30 seconds and make 15 LLM calls. Another run might take 4 hours and make 300 calls. The cost difference between these runs can be orders of magnitude.

A provider-level monthly threshold does not, by itself, express: "This run may consume at most $5 of the application budget." Your application or gateway needs a run identity and an enforcement rule for that boundary.

Without that finer-grained boundary, a single runaway run can consume a significant share of a broader provider allowance even when the provider control behaves exactly as documented.

### Provider identities may not match application tenants

Provider controls may apply at an organization, project, workspace, cloud-project, model, or key scope. Those scopes can help isolate workloads, but they do not automatically map shared credentials to your application's customer, workflow, run, or tool identities.

If a multi-tenant platform sends every customer's agent traffic through one shared provider identity, the provider cannot infer the application's per-customer budgets. A team can create separate provider projects, workspaces, or keys where supported, but that mapping is an application design choice rather than an automatic tenant model.

For example, consider an illustrative platform with 50 tenants sharing one provider project and a $50,000 soft monthly threshold. If one tenant's agent consumes $8,000, provider reporting can attribute the spend to the shared project but cannot infer the platform's tenant boundary unless the platform supplies a distinct provider identity or enforces that boundary elsewhere.

### Delayed enforcement

Budget dashboards, billing exports, and alerts are not the same as an in-process admission decision. Providers document different reporting and cutoff behavior. OpenAI now distinguishes soft spend alerts from optional hard organization/project spend limits, and documents that hard-limit enforcement is not instantaneous.

If an application polls those reporting surfaces and reacts later, work can continue between the underlying usage and the application's response. Request-time rate limits or exhausted-credit checks are different controls and should not be described as post-hoc.

### No pre-execution check

Providers can reject a request at their own boundary because of rate, credit, or account limits. What they generally do not expose is an application-defined reserve-commit lifecycle that asks, "Does this tenant's current run have enough of this budget unit for the estimated operation?" and holds that amount while the work is in flight.

For calls instrumented through Cycles, the application reserves its submitted estimate before execution. If the reservation is denied and the caller honors that denial, that protected call is not sent. Traffic that bypasses the integration is outside this guarantee.

### No graceful degradation

When a hard provider limit rejects a request, the provider does not know which application-specific fallback is safe. The application can still route to another model, use a cache, or reduce work, but it must implement that policy.

Production systems need nuance:

- Switch to a cheaper model when budget is low
- Reduce context window size
- Skip optional enrichment steps
- Serve cached responses instead of live inference
- Degrade gracefully for low-priority workflows while keeping high-priority ones running

Cycles can return `ALLOW`, configured `ALLOW_WITH_CAPS`, or `DENY` for submitted operations. The caller must translate caps into behavior—such as a cheaper route or smaller context—and must authorize the action separately.

### Multi-provider blind spots

Most teams do not use a single LLM provider.

A typical production stack might include:

- OpenAI for GPT-4 and embeddings
- Anthropic for Claude
- Google for Gemini
- A local model for low-latency classification

Each provider tracks its own usage independently. None of them know about spend on the other providers.

A team that has budgeted $500 per day across all providers has no single place to enforce that limit. OpenAI knows about OpenAI spend. Anthropic knows about Anthropic spend. Neither knows the total.

Cycles can account for submitted estimates across providers when every relevant path is instrumented into the same budget. The budget boundary is defined by the application, not inferred from provider billing.

## Comparison

| | Provider controls | Cycles |
|---|---|---|
| **Granularity** | Vendor-dependent: project/workspace/account windows, credits, or quotas | Submitted operation against configured tenant and subject scopes |
| **Scope** | Provider organization, project, workspace, cloud project, model, or key | Tenant plus caller-supplied subject hierarchy |
| **Enforcement timing** | Vendor-dependent: soft alert, request-time quota, credit check, or billing action | Pre-execution reservation for instrumented work |
| **Multi-provider** | One provider's traffic and billing | Shared budget only when callers submit all relevant provider paths |
| **Degradation** | Provider rejection or provider-specific policy; application chooses fallback | `ALLOW`, configured `ALLOW_WITH_CAPS`, or `DENY`; caller applies caps |
| **Protocol** | Vendor-specific dashboard and API | Open protocol with reserve-commit-release lifecycle |
| **Concurrency handling** | Vendor-specific | Atomic reservation mutation across the matching Cycles budget scopes |
| **Per-tenant enforcement** | Possible when provider identities and policies map to application tenants | Tenant-scoped keys and caller-supplied subject scopes |
| **Retry awareness** | Provider billing and idempotency semantics vary | Reusing the same Cycles idempotency key and request body deduplicates the budget mutation |

## The delay problem in detail

Reporting delay matters when an application treats a dashboard, export, or alert as its enforcement loop.

Consider an application that reacts only to a hypothetical usage report delayed by 60 seconds. Its agent makes calls at one per second, each assumed to cost $0.10, and its application threshold is $100.

At second 1,000, the agent has spent $100. But the polled report reflects spend as of second 940, so the application has not reacted.

The agent makes 60 more calls before the cap catches up. That is $6 of overspend — a 6% overrun.

Now increase the call rate. Five calls per second, each costing $0.50. At the same 60-second delay, that is 300 calls and $150 of overspend on a $100 cap — a 150% overrun.

This is an illustrative property of that polling design, not a measured overrun or a claim about every provider limit.

An instrumented Cycles path instead reserves the submitted estimate before execution. The reservation mutation is atomic across the matching Cycles scopes; the caller still needs accurate estimates, consistent integration, and settlement of actual usage.

## When to use both

Provider caps and Cycles are not mutually exclusive. They serve as different layers of defense.

### Keep provider caps as a safety net

Provider account, credit, quota, and billing controls remain an independent line of defense. Which of them is a hard stop depends on the provider and configuration.

Configure those controls according to their documented semantics. Do not treat a soft budget alert as an absolute maximum.

### Use Cycles for operational control

Cycles can supply the operational budget decision for instrumented paths:

- Per-tenant limits that align with pricing tiers
- Per-workflow limits that prevent individual runs from spiraling
- Per-run limits that bound the cost of any single agent execution
- Configured caps that the application can map to degradation behavior

This layer supplies the budget decision; the application remains responsible for complete instrumentation, authorization, and fallback behavior.

### Defense in depth

The combination creates defense in depth:

1. **Cycles** handles configured budgets for instrumented operations with pre-execution checks.
2. **Provider controls** independently constrain or report the provider traffic they cover.

Multiple independent controls can catch different failure modes, but their exact guarantees come from their documented configuration—not from their position in this diagram.

## Migration path

Teams that currently rely on provider caps alone can adopt Cycles incrementally.

**Step 1: Shadow mode.** Deploy Cycles in shadow mode. It evaluates budget decisions but does not enforce or persist the dry-run result. Have the application log the result and compare it with what actually happened.

**Step 2: Validate.** Review the shadow mode data. Are the budget allocations correct? Are the scope hierarchies right? Would enforcement have blocked legitimate work? Adjust the configuration.

**Step 3: Enforce on new workflows.** Enable enforcement for new or low-risk workflows first. Keep shadow mode on everything else.

**Step 4: Expand enforcement.** Gradually move more workflows from shadow mode to enforcement as confidence builds.

**Step 5: Recheck provider controls.** Confirm that alerts, credits, quotas, and billing actions still match the organization's independent safety requirements.

The practical result is layered control: keep the provider-native protections that fit your account, and add application-scoped budgets where provider identities and windows do not express the boundary you need.

## Sources

Provider behavior was rechecked on July 24, 2026:

- [OpenAI spend limits](https://developers.openai.com/api/docs/guides/spend-limits)
- [OpenAI projects and limits](https://help.openai.com/en/articles/9186755-managing-projects-in-the-api-platform)
- [OpenAI prepaid billing](https://help.openai.com/en/articles/8264644-what-is-prepaid-billing)
- [Anthropic API billing and usage credits](https://support.anthropic.com/en/articles/8977456-how-do-i-pay-for-my-api-usage)
- [Anthropic cost and usage reporting](https://support.anthropic.com/en/articles/9534590-cost-and-usage-reporting-in-console)
- [Google Cloud generative AI throughput quota](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/resources/throughput-quota)
- [AWS Bedrock service quotas](https://docs.aws.amazon.com/bedrock/latest/userguide/quotas.html)

## Next steps

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Try the [End-to-End Tutorial](/quickstart/end-to-end-tutorial) — zero to a working budget-guarded LLM call in ten minutes
- [Multi-Tenant AI Cost Control](/blog/multi-tenant-ai-cost-control-per-tenant-budgets-quotas-isolation) — how provider and application scopes differ in shared systems



# Cycles vs Rate Limiting: Why Velocity Controls Fail for AI Agents

Rate limiting is one of the most widely deployed control patterns in software.

It works. It has worked for decades.

But it was designed for a different problem than the one AI agents create.

Rate limiters answer: **how fast?**

Cycles answers: **how much?**

That distinction determines whether your system can burn through $10,000 overnight while staying perfectly within its RPM limit.

> **Run the numbers for your workload:** [Cost Calculator →](/calculators/claude-vs-gpt-cost-standalone) — rate limits do not bound spend; the calculator shows what one un-budgeted runaway loop is worth at your token rate.

## What rate limiting does well

Rate limiters are effective at three things.

### Abuse prevention

A rate limiter keeps a bad actor from hammering your API. It sets a ceiling on request velocity per caller, per endpoint, or per time window. That is essential for any public-facing service.

### Traffic shaping

Rate limiters smooth bursty traffic. They protect downstream services from sudden spikes, keep queue depths manageable, and help maintain latency targets under load.

### Fairness

In multi-tenant systems, rate limiters ensure one tenant cannot monopolize shared resources. Every caller gets a fair share of throughput.

These are real, valuable properties. Nothing in this article suggests removing your rate limiter.

The question is whether rate limiting alone is sufficient when autonomous agents enter the picture.

It is not.

## Where rate limiting fails for AI agents

AI agents break the assumptions that make rate limiting sufficient.

### Rate limiters do not track cumulative cost

A rate limiter knows how many requests passed through in the last minute. It does not know how much those requests cost in total.

An agent that makes 10 requests per minute stays within a 60 RPM limit. But if each request triggers a long-context GPT-4 call with tool use, the cost per request might be $0.50 or more. That is $300 per hour. $7,200 per day. All within the rate limit.

The rate limiter sees normal traffic. The bill tells a different story.

### Rate limiters cannot distinguish cheap calls from expensive calls

To a rate limiter, every request is identical. A call that uses 100 input tokens and a call that uses 100,000 input tokens count the same: one request.

This is the fundamental mismatch. AI workloads have extreme cost variance between requests. A simple classification call might cost $0.001. A multi-turn agentic workflow with tool calls might cost $5.00. Both are one request.

Rate limiting treats them identically. Runtime authority cannot afford to.

### Rate limiters have no per-run or per-workflow awareness

A rate limiter operates at the connection level. It does not know that five requests belong to the same agent run, or that a workflow has fanned out into twelve parallel sub-tasks.

It cannot enforce: "this workflow may only spend $2 total." It can only enforce: "this caller may make N requests per time window."

That means an agent can spawn sub-tasks, retry failed steps, and loop through tool calls — all within the rate limit — while the total cost of a single run spirals.

### An agent can stay within RPM limits and burn $10K overnight

This is not a theoretical risk. It is the most common failure mode teams report.

The agent is well-behaved. It respects rate limits. It does not spike. It does not look like abuse.

It simply runs continuously, making steady, moderately expensive calls. Each call is allowed. The total is not governed.

By morning, the bill is $10,000. Nothing in the rate limiter flagged it.

The problem is not velocity. The problem is unbounded cumulative spend.

### No graceful degradation

When a rate limiter triggers, it returns 429 Too Many Requests. The client backs off and retries.

That is a binary response: allowed or throttled.

AI agents need a richer vocabulary. Sometimes the right answer is not "stop" but "continue with a cheaper model." Or "reduce the number of tool calls." Or "skip the optional enrichment step."

Rate limiters cannot express these nuances. They have one lever: velocity.

## Comparison

| | Rate Limiter | Cycles |
|---|---|---|
| **Controls** | Request velocity (RPM, RPS) | Total budgeted exposure (cost, tokens, units) |
| **Granularity** | Per-caller, per-endpoint, per-time-window | Per-tenant, per-workspace, per-workflow, per-agent |
| **Cost-aware** | No — every request counts equally | Yes — reserves estimated cost, commits actual cost |
| **Pre-execution budget check** | Velocity only — no cumulative awareness | Yes — checks remaining budget across all scopes before execution |
| **Concurrency-safe** | Yes for velocity counting | Yes — atomic reservations prevent race conditions on budget |
| **Degradation support** | No — binary allow/throttle | Yes — three-way decision: ALLOW, ALLOW_WITH_CAPS, DENY |

## How Cycles works where rate limiting cannot

Cycles introduces a reserve-then-commit model that is fundamentally different from velocity counting.

Before an agent action executes:

1. The system declares how much budget the action is expected to consume.
2. Cycles checks whether that budget is available across all applicable scopes (tenant, workspace, workflow, agent).
3. If available, the budget is atomically reserved. No other concurrent request can claim the same budget.
4. The action executes.
5. After execution, the system commits the actual cost. If the actual cost is less than the reservation, the remainder is released automatically.

This model answers questions that rate limiters cannot:

- Has this run already consumed too much? Then deny or degrade the next step.
- Is the tenant approaching its daily limit? Then switch to a cheaper model.
- Are concurrent requests about to exceed the workflow budget? The reservation is atomic — only one will succeed.
- Did the action cost less than expected? The unused budget is released for other work.

## The concurrency problem

Rate limiters handle concurrency well for velocity. They are designed for it.

But budget governance under concurrency is a different problem.

Consider two agent threads running in parallel against the same workflow budget. The budget has $5 remaining. Both threads check the budget, both see $5 available, and both proceed. Total spend: $10 against a $5 budget.

This is a classic time-of-check-to-time-of-use (TOCTOU) race condition. Rate limiters do not protect against it because they do not track cumulative spend.

Cycles handles this with atomic reservations. When the first thread reserves $5, that budget is immediately unavailable to the second thread. The second thread's reservation attempt sees the reduced balance and can be denied or degraded.

No oversubscription of the submitted estimates across matching Cycles ledgers. Actual provider spend still depends on estimate quality, complete instrumentation, and settlement policy.

## When to use both together

Rate limiting and Cycles solve different problems. Most production systems should use both.

**Keep your rate limiter for:**

- Abuse prevention — stopping bad actors from flooding your API
- Traffic shaping — smoothing bursts to protect downstream services
- Fairness — ensuring no single caller monopolizes throughput
- DDoS mitigation — absorbing malicious traffic spikes

**Add Cycles for:**

- Budget governance — bounding total spend per tenant, workflow, or run
- Cost-aware decisions — distinguishing cheap calls from expensive ones
- Graceful degradation — downgrading to cheaper models when budget is low
- Pre-execution enforcement — stopping expensive work before it starts
- Concurrency-safe accounting — preventing race conditions on budget

The two sit at different points in the request path.

A rate limiter typically sits at the edge — at the API gateway or load balancer. It decides whether the request may enter the system at all.

Cycles sits inside the application logic — at the point where an agent is about to make an expensive decision. It decides whether that specific action is allowed given the current budget state.

A request can pass the rate limiter (it is within velocity limits) and still be denied by Cycles (the budget is exhausted). These are independent, complementary checks.

## The architecture in practice

A typical flow looks like this:

```
Request arrives
    → Rate limiter: within RPM? → Yes → proceed
    → Cycles: budget available? → Reserve
        → Execute agent action (LLM call, tool use)
        → Cycles: commit actual cost, release remainder
```

If the rate limiter says no, the request never reaches Cycles. That is correct — abuse prevention should happen first.

If the rate limiter says yes but Cycles says no, the agent action does not execute. That is also correct — the work is within velocity limits but exceeds budget.

If both say yes, the action proceeds with reserved budget. After execution, the actual cost is committed and any unused reservation is released.

## The key insight

Rate limiting and runtime authority are orthogonal controls.

Rate limiting governs the speed of requests. It prevents bursts and abuse. It is stateless in the sense that it does not track what those requests cost in aggregate.

Runtime budget authority governs submitted cumulative exposure. It is stateful — it tracks reservations, commits, and remaining balances across configured scopes.

An AI agent that respects rate limits can still create unbounded cost.

For paths that must reserve before execution, Cycles prevents concurrent submitted estimates from oversubscribing the matching ledgers. Actual usage can exceed an estimate and is handled by the configured commit-overage policy; bypass traffic remains outside the boundary.

Rate limiting answers **how fast?**

Cycles answers **how much?**

Production systems need both answers.

## Next steps

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Try the [End-to-End Tutorial](/quickstart/end-to-end-tutorial) — zero to a working budget-guarded LLM call in ten minutes
- [The True Cost of Uncontrolled AI Agents](/blog/true-cost-of-uncontrolled-agents) — what happens when rate limits are the only line of defense



# Exposure: Why Rate Limits Leave Agents Unbounded

Exposure is the total cost, risk, or damage an autonomous system can create before something stops it.

It is not the same as spend. In an illustrative scenario, 200 mistaken customer emails cost about $1.40 in model tokens while their unquantified customer and business impact can be much larger. Low spend does not imply low exposure.

> **Quantify exposure for your agent:** [Blast Radius Risk Calculator →](/calculators/ai-agent-blast-radius-standalone) — model action classes by reversibility and visibility; the catastrophic *irreversible + public* class is what rate limits leave unbounded.

## Why exposure matters

Every autonomous system has two numbers:

1. **Spend** — what it costs to run (tokens, compute, API fees)
2. **Exposure** — what it can do before it is stopped (emails sent, records modified, deploys triggered, dollars committed)

Most cost controls target one dimension. Rate limits cap throughput. Provider controls may alert on budgets, consume prepaid credits, or enforce account and capacity limits. Observability dashboards report what happened. None automatically represents every application run, tenant, or side effect.

None of these bound exposure, because none of them enforce limits **before** the next action executes.

## Rate limits don't help

A rate limit of 100 requests per minute does not prevent an agent from sending 100 emails in that minute. It controls velocity, not authorization. The agent is never asked "should this action proceed?" — it is only told "slow down."

Rate limits are designed for shared infrastructure protection. They are not designed for autonomous agent governance. See [Why Rate Limits Are Not Enough](/concepts/why-rate-limits-are-not-enough-for-autonomous-systems).

## Observability doesn't help

Dashboards and tracing systems record what happened. They can alert after the fact. But by the time a human sees the alert, the agent has already acted. In the email scenario, the 200 messages are sent. In a runaway loop, the budget is already burned.

Observability is essential — but it observes exposure. It does not bound it. See [From Observability to Enforcement](/concepts/from-observability-to-enforcement-how-teams-evolve-from-dashboards-to-budget-authority).

## How reserve-commit bounds exposure

Cycles bounds exposure by requiring agents to **reserve** budget before execution and **commit** the actual cost afterward.

The reservation is the enforcement point. If sufficient budget is unavailable, a live reservation fails and correctly integrated protected work does not execute. This bounds the submitted cumulative exposure at that boundary; it does not cap all possible business harm, validate the estimate, or cover work that bypasses instrumentation.

This applies to both financial exposure (USD, tokens) and operational exposure (risk points). A toolset-scoped budget denominated in RISK_POINTS can cap the number of consequential actions an agent takes, regardless of their dollar cost. See [Action Authority](/concepts/action-authority-controlling-what-agents-do).

For practical strategies on sizing reservations and estimating exposure before execution, see [Exposure Estimation](/how-to/how-to-estimate-exposure-before-execution-practical-reservation-strategies-for-cycles).

## Next steps

- [Glossary: Exposure](/glossary#exposure) — formal definition
- [Runaway Agents and Tool Loops](/incidents/runaway-agents-tool-loops-and-budget-overruns-the-incidents-cycles-is-designed-to-prevent) — what unbounded exposure looks like in practice
- [Demos](/demos/) — the runaway agent demo shows a cost runaway stopped at $1.00 by reserve-commit



# From Observability to Enforcement: How Teams Evolve from Dashboards to Runtime Authority

Most teams do not begin with enforcement.

They begin with visibility.

They add logs.  
They add traces.  
They monitor provider usage.  
They build dashboards.  
They set alerts for abnormal spend.  
They review incidents after they happen.

That is the right starting point.

But as autonomous systems become more capable, visibility alone stops being enough.

> **Quantify the gap dashboards leave open:** [Cost Calculator →](/calculators/claude-vs-gpt-cost-standalone) · [Blast Radius Risk Calculator →](/calculators/ai-agent-blast-radius-standalone) — observability records what happened; the calculators show the budget envelope and the risk envelope that no alert closes.

At some point, the question changes from:

::: info
What happened?
:::

to:

::: info
What should be allowed to happen next?
:::

That is the transition from observability to enforcement.

It is also the transition that Cycles is designed to support.

## Why observability comes first

Observability is usually the first control layer because it is easy to adopt and low risk.

It does not block execution.  
It does not change application behavior.  
It does not require the team to make hard policy decisions immediately.

It helps answer questions like:

- which workflows are expensive?
- which tenants consume the most?
- where do retries happen?
- which tools are called most often?
- which runs are unusually long?
- how does actual usage vary over time?

These are necessary questions.

A team cannot govern what it cannot see.

That is why most systems start here.

## Why observability eventually stops being enough

The problem is that observability is passive.

It can explain what happened.  
It cannot, by itself, stop the next incident.

A dashboard can tell you a workflow burned through budget.  
An alert can tell you a tenant exceeded expected usage.  
A trace can show you that a tool loop retried six times.

But all of that happens after the relevant work already executed.

That matters less in traditional software where failures are often discrete and bounded.

It matters much more in autonomous systems, where cost and side effects accumulate over time.

The more a system can:

- loop
- retry
- fan out
- recurse
- continue in the background
- trigger side effects

the less sufficient post-hoc visibility becomes.

## The maturity curve

Most teams move through a recognizable sequence.

### Stage 1: Basic usage visibility

At this stage, teams can answer:

- how much did we spend?
- which provider was used?
- which tenant generated the most traffic?

This is useful, but still coarse.

### Stage 2: Workflow-level observability

The team begins to understand:

- which workflows are expensive
- how usage distributes across runs
- where retries cluster
- which tools amplify cost
- which execution paths are noisy

This is much better.

The team now has operational visibility into autonomous behavior, not just aggregate billing.

### Stage 3: Alerting and anomaly detection

Next, the team starts reacting to:

- usage spikes
- tenant anomalies
- unexpectedly long runs
- retry storms
- sudden workflow fan-out

This creates faster feedback, but still does not introduce bounded control.

### Stage 4: Soft controls and heuristics

Teams often add:

- ad hoc loop counters
- static max-step thresholds
- timeout tuning
- hardcoded fallbacks
- kill switches
- per-feature caps

These controls can help, but they are often fragmented and inconsistent.

### Stage 5: Runtime authority

Eventually the team realizes it needs one thing the earlier stages do not provide:

**a runtime decision point before autonomous work proceeds**

That is where enforcement begins.

## What changes at the enforcement stage

Enforcement adds a new question:

::: info
Is this action still authorized to continue under the current budget?
:::

That is different from asking how much the system spent yesterday or which run was expensive.

It means the system now needs to make bounded decisions in real time.

That includes questions like:

- may this model call proceed?
- should this tool invocation still be allowed?
- is this run already too expensive?
- should the workflow degrade instead of continuing normally?
- has this tenant exhausted its budget envelope?
- should a background job stop here?

This is the move from descriptive operations to governing execution.

## Why dashboards are necessary but insufficient

A useful way to think about it is:

- **dashboards explain**
- **runtime authority decides**

You still want the dashboard.

You still want traces and alerts.

But once systems become autonomous, decision-making needs a control surface too.

Otherwise teams end up in a loop of:

1. observe incident
2. write another heuristic
3. observe a new variant
4. add another exception
5. repeat

That tends to produce fragile policy and unclear ownership.

## The missing primitive

The missing primitive is not “more analytics.”

It is a way for the runtime to ask for bounded room to act before work proceeds.

That is what Cycles introduces.

At a high level:

1. declare intended exposure
2. reserve budget
3. execute
4. commit actual usage or release the remainder

This turns enforcement into a lifecycle rather than a spreadsheet.

Instead of only knowing what happened later, the system can decide whether work is allowed to continue now.

## Why this transition matters more for autonomous systems

Autonomous systems create a different operational shape than traditional request-response applications.

A single initiating event may lead to:

- many model calls
- repeated retries
- multiple tool invocations
- workflow branching
- asynchronous continuation
- external side effects

That means “request count” and “API throughput” stop being good proxies for real operational exposure.

The system needs to reason about total bounded execution, not just traffic volume.

This is why observability naturally leads to runtime authority as systems mature.

## A common evolution pattern

A typical team often evolves like this.

### Early phase

The team is mostly trying to understand behavior.

It wants visibility, not restrictions.

Questions sound like:

- what is this costing?
- where are tokens going?
- which paths are expensive?

### Middle phase

The team starts seeing incidents or near misses.

Questions become:

- why did this run keep going?
- why did retries multiply spend?
- why did one tenant consume so much?
- why did this workflow call tools so many times?

At this point, observability exposes the problem clearly.

### Later phase

The team realizes that understanding is not the same as control.

Questions become:

- how do we stop this next time?
- how do we prevent it before cost lands?
- how do we add hard boundaries without breaking everything?
- how do we make policy hold under retries and concurrency?

That is when runtime authority becomes necessary.

## What enforcement should not mean

Enforcement does not have to mean “deny everything aggressively.”

A mature control model often includes multiple outcomes:

- allow normally
- return a hypothetical dry-run decision for the application to log
- degrade to a smaller model
- disable costly tools
- switch to read-only behavior
- reduce concurrency
- deny further execution

This is important because the move from observability to enforcement is not a move from flexibility to rigidity.

It is a move from passive awareness to intentional control.

## Why teams get stuck before enforcement

Many teams understand the value of bounded execution but still hesitate to adopt it.

Common reasons include:

- uncertainty about the right thresholds
- fear of breaking production
- incomplete understanding of workflow usage
- lack of estimate quality
- worry about false denials
- fragmented ownership between platform and application teams

These are real concerns.

That is why Cycles includes shadow-mode-friendly thinking as part of the model.

Teams often need to observe policy against real workloads before turning on hard stops.

## The role of shadow mode in the maturity curve

Shadow mode is often the bridge between observability and enforcement.

With `dry_run: true`, Cycles returns a hypothetical decision without creating a reservation or modifying balances. The application must retain that response and correlate it with actual usage. Those records let teams ask:

- what would have been denied?
- which runs would have exceeded budget?
- which tenants are routinely near limits?
- how well do estimates match actuals?
- what should degrade instead of fail?

That means the maturity curve is usually not:

**observe → enforce**

It is more often:

**observe → evaluate in shadow → enforce intentionally**

That is a much safer operational path.

## What changes once runtime authority exists

Once a system has a real runtime authority, several things change.

### Instrumented incidents become easier to bound

Instead of relying only on dashboards and operator reaction, a mandatory application boundary can stop or degrade protected work when a budget request fails.

### Policy becomes explicit

Instead of scattered budget heuristics across code, the platform gains a clearer model of submitted exposure at tenant and workflow levels. An application can use a run ID as `subjects.workflow` when it needs one ledger per execution; a run ID in `dimensions` is attribution only.

### Teams can reason about autonomy operationally

The conversation changes from:

- “why did this get expensive?”

to:

- “what execution envelope should this class of work have?”

That is a more mature operational question.

### Platform and product alignment improves

Budgets become a shared boundary between:

- platform economics
- product behavior
- execution safety

That is healthier than leaving those concerns disconnected.

## A concrete example

Imagine a support automation platform.

At first, the team only tracks:

- provider cost
- request count
- tenant usage totals

Then it starts seeing specific incidents:

- some runs call models repeatedly
- some workflows retry several times
- some tenants concentrate usage in a few expensive flows

The team adds dashboards and alerts. That helps explain the incidents.

But then one run again consumes far too much budget overnight.

At that point, the missing piece is clear.

The platform does not need another chart.  
It needs a way to say:

- protected actions in this run share a workflow ledger with this allocation
- this tenant has this much remaining
- this application should degrade the workflow when a reservation is denied or returned caps require it
- this next action may not proceed unless budget is reserved first

That is the shift from observability to enforcement.

## Why Cycles fits this transition

Cycles is designed for teams that have already realized visibility alone is not enough.

It provides a runtime model for:

- pre-execution budget checks
- reserve → commit / release lifecycle handling
- hierarchical scope enforcement
- retry-safe accounting
- shadow evaluation before hard enforcement

In other words, it helps teams operationalize what observability has already taught them.

## Summary

Observability is where autonomous control begins.

It helps teams understand:

- what systems are doing
- where cost accumulates
- which workflows are unstable
- where retries and fan-out create pressure

But understanding alone does not prevent the next incident.

As systems become more autonomous, teams need a control layer that can decide whether work should continue before cost and side effects grow further.

That is the move from dashboards to runtime authority.

That is the move Cycles is built to support.

## Next steps

To explore the Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)
- [AI Agent Cost Management: The Complete Guide](/blog/ai-agent-cost-management-guide) — the five-tier maturity model for moving from no controls to hard enforcement
- [Cycles vs LLM Proxies and Observability Tools](/blog/cycles-vs-llm-proxies-and-observability-tools) — where budget enforcement fits alongside LiteLLM, Portkey, Helicone, and Langfuse



# How Cycles Compares to Rate Limiters, Observability, Provider Caps, In-App Counters, and Job Schedulers

## Quick comparison

| Approach | What it controls | Pre-execution? | Per-tenant? | Cost-aware? | Degradation? |
|---|---|:---:|:---:|:---:|:---:|
| **Rate limiter** | Request velocity | Velocity only | Partial | No | No |
| **Observability** | Traces, metrics, and product-specific controls | Product/config dependent | Product/config dependent | Usually | Product/config dependent |
| **Provider controls** | Vendor spend, credits, or capacity | Soft or hard, vendor dependent | Provider identity only | Yes for covered usage | Application chooses fallback |
| **In-app counter** | Custom metric | Partial | Partial | Partial | No |
| **Job scheduler** | Execution timing | No | No | No | No |
| **Cycles** | Bounded budget, risk exposure | Yes | Yes | Yes | Yes (ALLOW / ALLOW_WITH_CAPS / DENY) |

---

Teams building autonomous systems usually already have some controls in place.

- Rate limiters.
- Observability platforms.
- Provider budget caps.
- In-app usage counters.
- Job schedulers with retry logic.

These are all reasonable tools. They each solve real problems.

But none of them solve the problem Cycles is designed for: **governing bounded execution before autonomous work proceeds**.

This article walks through each alternative, explains what it does well, where it falls short, and how Cycles differs. For the short-form essay version of the same argument — focused on *why* tool-local controls cannot stretch to cover an agent that spans providers, tools, tenants, and workers — see [Agents Are Cross-Cutting. Your Controls Aren't.](/blog/agents-are-cross-cutting-your-controls-arent).

## Rate limiting vs Cycles

Rate limiters control **velocity**.

They answer: how many requests per second, minute, or hour may this caller make?

That is useful for:

- abuse prevention
- traffic shaping
- fairness across tenants
- protecting downstream systems from bursts

### Where rate limiting falls short

Rate limiters do not track total consumption.

An agent can stay within its request-per-second limit and still burn through an entire budget over hours.

Nothing spikes.
Nothing looks like abuse.
The system is simply allowed to continue indefinitely.

Rate limiters also do not understand execution context:

- they do not know this is the third retry of the same action
- they do not know the run is already 80% over budget
- they do not know the workflow has fanned out into expensive sub-tasks
- they do not distinguish between a $0.001 call and a $2.00 call

Every request looks the same to a rate limiter.

### How Cycles differs

Cycles controls **total bounded exposure**, not request velocity.

Before work begins, the system reserves budget.
After work completes, it commits actual usage.
Unused budget is released.

That means a run cannot quietly accumulate cost beyond its envelope, regardless of how slowly or quickly it acts.

| | Rate limiter | Cycles |
|---|---|---|
| Controls | Requests per time window | Total budgeted exposure |
| Granularity | Per-caller or per-endpoint | Per-tenant, workspace, app, workflow, agent, toolset |
| Understands retries | No | Yes (idempotent reservations) |
| Understands cost | No | Yes (reserve estimated, commit actual) |
| Pre-execution check | Velocity only | Budget availability across scopes |
| Lifecycle | Stateless counter | Reserve → execute → commit/release |

**Keep your rate limiter.**

It protects against bursts and abuse.

But do not expect it to govern what an autonomous system is allowed to consume in total.

## Observability vs Cycles

Observability answers: **what happened?**

It helps teams understand cost, behavior, and anomalies after execution occurs.

Good observability includes:

- usage dashboards
- per-tenant cost breakdowns
- workflow traces
- retry and error distributions
- spend-over-time charts
- anomaly alerts

### Where observability falls short

Observability is passive.

A dashboard can show that a runaway workflow consumed $400 overnight.
An alert can tell you a tenant exceeded expected usage.
A trace can reveal a tool loop that retried twelve times.

All of that is valuable.

None of it prevented the incident.

Post-hoc visibility helps teams improve.
It does not help the runtime decide whether the next action should proceed.

The gap is especially visible in autonomous systems where:

- loops can run for hours without triggering alerts
- cost accumulates gradually, not in spikes
- the damage is done by the time the alert fires
- response requires human intervention, which may not come fast enough

### How Cycles differs

Cycles introduces a **pre-execution decision point**.

Instead of only explaining what happened afterward, Cycles determines whether work is allowed to continue now.

The system asks: is there enough budget remaining for this action?

If yes, budget is reserved and work proceeds.
If no, the system can DENY, degrade (ALLOW_WITH_CAPS), or defer the action. When degrading, Cycles returns cap fields — `max_tokens`, `max_steps_remaining`, `tool_allowlist`, `tool_denylist`, and `cooldown_ms` — so the caller knows exactly how to constrain the next action.

| | Observability | Cycles |
|---|---|---|
| Timing | After execution | Before and during execution |
| Purpose | Explain what happened | Decide what may happen |
| Response to overruns | Alert, investigate, fix later | Deny, degrade, or defer in real time |
| Requires human response | Often yes | No (automated enforcement) |
| Lifecycle awareness | Traces and logs | Reserve → commit/release |

**Keep your observability stack.**

Cycles benefits from good observability. It does not replace it.

But do not confuse explaining the past with governing the present.

## Provider budget caps vs Cycles

Most LLM providers offer some form of spending cap or usage limit.

Depending on the vendor and plan, these include:

- monthly spend alerts or hard limits on an organization or project
- prepaid-credit cutoffs
- daily or monthly spend alerts
- per-model request or token-rate limits

### Where provider caps fall short

Provider controls are scoped to vendor-defined identities and are external to your application's own subject hierarchy.

They operate at the wrong level of granularity for autonomous systems.

**No automatic application-tenant enforcement.**
A provider project, workspace, account, or key can isolate traffic assigned to it. When multiple application tenants share that identity, however, the provider cannot infer their separate ledgers. One tenant can consume the shared allowance.

**No automatic per-run or per-workflow limit.**
A team could dedicate a provider project or key to a workload where supported, but a shared provider boundary does not learn the application's workflow or run identity.

**Provider-specific enforcement.**
Some controls only alert; others reject affected requests when a credit, quota, or hard spend limit binds. The provider does not know which application-specific degradation path is safe, so the application must choose the fallback.

**No application reserve-commit semantics.**
Provider controls can reject at their request boundary, but they do not expose a reserve-commit lifecycle for an application-defined tenant/run estimate. For example, OpenAI documents that hard spend-limit enforcement is not instantaneous and tracked spend can slightly exceed the configured amount.

**Timing varies.**
Alerts, dashboards, hard spend limits, credit checks, and rate quotas have different timing. Treat each according to its documented behavior rather than assuming every control is either immediate or delayed.

**No lifecycle awareness.**
Provider controls do not infer that a request belongs to an application retry, how much a particular run has used, or which workflow-specific degradation is safe.

### How Cycles differs

Cycles provides budget state for caller-supplied application scopes.

It operates at the level your system actually needs:

- per-tenant
- per-workspace
- per-app
- per-workflow
- per-agent
- per-toolset

Budget is reserved before execution rather than inferred after usage occurs.

| | Provider controls | Cycles |
|---|---|---|
| Scope | Vendor organization, project, workspace, account, key, credit, or quota | Caller-supplied tenant, workspace, app, workflow, agent, toolset |
| Enforcement | Soft or hard, vendor/configuration dependent | Live reservation accepted with optional configured caps, or rejected |
| Timing | Alert, request-time rejection, or non-instantaneous spend cutoff | Pre-execution hold for instrumented work |
| Multi-tenant aware | Only through explicit provider-identity mapping | Yes, when the caller submits tenant scopes |
| Degradation support | Provider-specific; application chooses fallback | Configured caps returned; application applies fallback |
| Retry-safe | Provider-specific | Same idempotency key and body deduplicate a budget mutation |
| Under your control | No (vendor-managed) | Yes (self-hosted, operator-defined) |

**Provider controls are an independent safety layer.**

Depending on their semantics, they can alert, constrain capacity, exhaust credits, or stop provider traffic at an organization or project boundary.

But they are not a substitute for application-level budget governance.

## In-app counters vs Cycles

Many teams build their own usage counters.

These are typically:

- a database column tracking tokens used per tenant
- an in-memory counter incremented after each model call
- a Redis key tracking spend per run
- a custom middleware that checks a threshold before calling the model

This is often the first thing teams build when they realize they need per-tenant or per-run limits.

### Where in-app counters fall short

In-app counters work in simple cases. They break down as systems become more complex.

**Race conditions under concurrency.**
If two requests check the counter simultaneously, both may see "under budget" and proceed. The result is overspend. Solving this correctly requires atomic operations, locks, or compare-and-swap — which most ad hoc counters do not implement.

**No reservation semantics.**
Counters typically increment after execution. That means the system commits to work before knowing whether the budget can absorb it. If the model call costs more than expected, the counter reflects reality too late.

**No hierarchical scopes.**
A counter per tenant is useful. But autonomous systems often need limits at multiple levels such as tenant, workspace, app, workflow, agent, and toolset. An application can key a workflow ledger per run, while action authorization remains a host concern. Building and maintaining hierarchical counters with correct rollup logic is significantly more complex than a single counter.

**Fragile under retries.**
If a request fails and retries, does the counter increment once or twice? If the retry uses a different code path, does it check the same counter? Most ad hoc counters do not handle retries cleanly.

**No lifecycle management.**
Counters have no concept of reservations, releases, TTLs, or grace periods. Budget is either "used" or "not used." There is no way to reserve estimated exposure before execution, commit actuals afterward, or release unused budget.

**Scattered implementation.**
Counter logic often lives inside business code, spread across services and endpoints. It is hard to audit, hard to test, and hard to make consistent.

### How Cycles differs

Cycles replaces ad hoc counters with a purpose-built runtime authority.

It handles concurrency, retries, hierarchical scopes, and lifecycle semantics as first-class concerns — not afterthoughts.

| | In-app counter | Cycles |
|---|---|---|
| Concurrency safety | Usually racy | Atomic reservations |
| Timing | Post-execution increment | Pre-execution reservation |
| Hierarchical scopes | Rarely | Built-in (tenant → workspace → app → workflow → agent → toolset) |
| Retry handling | Fragile | Idempotent lifecycle |
| Lifecycle support | None | Reserve → commit / release / extend |
| TTL and expiry | Manual if at all | Built-in reservation TTL and grace |
| Audit and consistency | Scattered | Centralized authority |

**In-app counters are a natural starting point.**

But they tend to accumulate correctness bugs and edge cases as the system scales.

Cycles is what teams adopt when ad hoc counters stop being reliable under real concurrency, retries, and fan-out.

## Job schedulers and retry logic vs Cycles

Job schedulers manage **when and how work executes**.

They handle:

- task queuing
- retry policies (backoff, max attempts)
- cron-based scheduling
- dead-letter queues
- concurrency limits on workers
- task deduplication

Common examples include Celery, Sidekiq, Bull, Temporal, Spring Batch, and Quartz.

### Where job schedulers fall short

Job schedulers govern execution mechanics.
They do not govern execution economics.

**Retry policies do not understand budget.**
A scheduler may retry a failed task five times. Each retry may call an LLM. The scheduler does not know or care that the run has already exhausted its budget. It only knows that the retry count has not been reached.

**Concurrency limits are not budget limits.**
A scheduler may allow ten concurrent workers. That limits parallelism, not total cost. Ten workers can each burn through expensive model calls simultaneously without any aggregate budget check.

**No cost awareness.**
A scheduler does not know that one task costs $0.01 and another costs $5.00. It treats all tasks equally. It cannot route a task to a cheaper path when budget is low, or deny an expensive task when the run is nearly exhausted.

**No cross-run or cross-tenant visibility.**
A scheduler manages individual jobs. It does not maintain a budget ledger across tenants, workflows, or time windows. It cannot answer: "has this tenant already consumed too much today?"

**Scheduling is orthogonal to governance.**
A scheduler decides: should this task run now, later, or again?
It does not decide: is this task allowed to run given the current budget state?

### How Cycles differs

Cycles provides budget governance that is orthogonal to — and complementary with — job scheduling.

A scheduler can call Cycles before executing a task.
Cycles can tell the scheduler whether the task should proceed, degrade, or be denied.

| | Job scheduler | Cycles |
|---|---|---|
| Controls | When and how work runs | Whether work is allowed given budget |
| Retry awareness | Max attempts, backoff | Budget remaining across retries |
| Cost awareness | None | Reserve estimated, commit actual |
| Scope | Per-job or per-queue | Per-tenant, workspace, app, workflow, agent, toolset |
| Degradation | Not built-in | Three-way (ALLOW / ALLOW_WITH_CAPS / DENY) |
| Cross-tenant limits | No | Yes |
| Complements | Execution engine | Runtime authority |

**Keep your scheduler.**

It is the right tool for managing execution timing and retry mechanics.

But pair it with a runtime authority so retries and fan-out do not become unbounded cost.

## Capability matrix

The table below maps specific capabilities against each approach.

| Capability | Rate limiter | Observability | Provider cap | In-app counter | Job scheduler | Cycles |
|---|---|---|---|---|---|---|
| Pre-execution budget check | No | No for tracing-only tools | Yes, at provider/gateway scope | Partial | No | Yes |
| Caller-estimated reserve-commit lifecycle | No | No | No | Partial, if custom-built | No | Yes |
| Per-tenant limits | Partial | Attribution, not enforcement | Partial, product-specific | Partial | No | Yes |
| Per-workflow / per-agent limits | Partial, with custom keys | Attribution, not enforcement | Partial, product-specific | Partial | Partial | Yes |
| Hierarchical scopes | No | Partial for grouping | Partial, product-specific | Partial | Partial | Yes |
| Cost-aware decisions | No | No for tracing-only tools | Yes | Partial | No | Yes |
| Retry / idempotency safety | Partial | No | Partial | Partial | Yes for job execution | Yes for budget lifecycle |
| Configured `ALLOW_WITH_CAPS` outcome | No | No | Product-specific alternatives | Partial | No | Yes; caller applies caps |
| Concurrency-safe accounting | No | No | Product-specific | Partial | Partial | Yes, for reservations |
| Real-time enforcement | Yes, for traffic | No for tracing-only tools | Yes, at provider/gateway scope | Partial | No | Yes, at instrumented boundary |
| Post-hoc analysis and traces | No | Yes | Partial | Partial | Partial | Partial; lifecycle records only |
| Traffic shaping / abuse prevention | Yes | No | Partial | No | Partial | No |
| Execution scheduling and retries | No | No | No | No | Yes | No |

A few things worth noting:

- No single tool covers every row. That is expected.
- "Partial" covers product-specific support or substantial custom work. Check the exact gateway, provider, or scheduler rather than treating a category as one uniform product.
- Cycles does not try to replace traffic shaping, observability, or scheduling. Those are separate concerns with mature tooling. Cycles focuses on the budget governance column because that is the gap most teams hit as autonomous systems scale.

Each of these tools earns its place in a production stack. The question is whether the stack has a gap where runtime authority should be.

## They work together

Cycles does not replace any of these tools.

It fills the gap between them.

A well-governed autonomous system typically includes:

- a **rate limiter** for traffic shaping and abuse prevention
- an **observability platform** for visibility, traces, and alerts
- **provider controls** as an independent vendor-side boundary
- a **job scheduler** for execution timing and retry policies
- **Cycles** as the budget authority for caller-submitted application scopes

These layers are complementary, not competitive.

The question is not "which one should I use?"

It is "which layer is missing?"

For most teams building autonomous systems, the missing layer is runtime authority.

## When to adopt Cycles

Consider adding Cycles to your stack when:

- **Agents run autonomously** — without human-in-the-loop approval for each action
- **Cost is unpredictable** — fan-out, tool loops, or retries make per-run cost hard to bound
- **Multiple tenants share infrastructure** — one tenant's runaway agent should not affect others
- **You need graceful degradation** — switching to cheaper models or reducing scope when budget is low, rather than hard-failing
- **Governance requires budget evidence** — lifecycle records showing which instrumented operations reserved, committed, released, or were denied against configured budgets
- **You've outgrown ad hoc counters** — custom counters work until concurrency, retries, and hierarchy make them unreliable

If none of these apply yet, start with [shadow mode](/how-to/shadow-mode-in-cycles-how-to-roll-out-budget-enforcement-without-breaking-production) to see what enforcement would look like on your current traffic.

## Next steps

To explore the Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)
- Browse the full [Integration Ecosystem](/how-to/ecosystem)
- [At-a-Glance Comparisons](/concepts/comparisons) — quick reference table comparing Cycles to seven alternatives
- [5 Real-World AI Agent Failures That Budget Controls Would Have Prevented](/blog/ai-agent-failures-budget-controls-prevent) — concrete failure scenarios and what each approach prevents
- [Cycles vs LLM Proxies and Observability Tools](/blog/cycles-vs-llm-proxies-and-observability-tools) — how Cycles complements LiteLLM, Portkey, Helicone, and Langfuse
- [Budget Wrapper vs Budget Authority](/blog/vibe-coding-budget-wrapper-vs-budget-authority) — why wrapping LLM calls is not the same as governing them
- [AI Agent Cost Management Guide](/blog/ai-agent-cost-management-guide) — comprehensive guide to managing AI agent costs at scale



# Idempotency, Retries, and Concurrency: Why Cycles Is Built for Real Failure Modes

Most budget systems look correct in the happy path.

A request arrives.  
The system checks a counter.  
Work executes.  
Usage is recorded.  
Everything looks fine.

Real systems do not behave that cleanly.

They retry.  
They time out.  
They crash halfway through execution.  
They send duplicate requests.  
They run multiple workers at once.  
They fan out across steps that all consume budget concurrently.

That is where naive accounting breaks down.

Cycles is built for these conditions on purpose.

It is not only a budgeting model.  
It is a runtime control model designed for failure, duplication, and concurrent execution.

## Why happy-path accounting is not enough

A simple usage counter can tell you how much was spent after work is done.

That may be enough for reporting.

It is usually not enough for safe enforcement.

Consider a few common failure cases:

- a client retries because it did not receive a response
- a worker crashes after reservation but before reconciliation
- two workers both try to reserve against the same remaining budget
- a duplicate message is processed twice
- a workflow branch commits usage after a parent path already retried
- actual usage arrives late, out of order, or more than once

If the accounting model is not designed for these cases, the system tends to produce one of three outcomes:

- accidental double-spend
- false denials caused by leaked reservations
- inconsistent enforcement under concurrency

These are not edge cases in autonomous systems.  
They are normal operating conditions.

## The problem with naive budget checks

A naive budget check often looks like this:

1. read current balance
2. compare against requested amount
3. if enough remains, proceed
4. update the balance later

That seems reasonable until two things happen at once.

For example:

- worker A reads available budget = 100
- worker B reads available budget = 100
- both decide to proceed
- both consume 80

Now the system has allowed 160 units of work against 100 units of available budget.

This is the classic race condition that appears whenever control decisions are separated from atomic state changes.

Cycles exists to avoid this category of failure.

## Why idempotency matters

Idempotency means the same logical action can be retried safely without being counted multiple times.

This is essential because retries happen for many reasons:

- the client timed out waiting for a response
- the network dropped after the server processed the request
- a worker crashed after partially completing work
- a message broker redelivered the same event
- an upstream service retried defensively

Without idempotency, every retry looks like a new request.

That can create:

- duplicate reservations
- duplicate commits
- duplicate releases
- budget drift
- over-counting that has nothing to do with real usage

In a production control plane, retry safety is not optional.  
It is part of correctness.

## Why reservation alone is not enough

Some systems try to solve budgeting with simple pre-checks or flat quota decrements.

That helps, but it is still incomplete.

Cycles uses a lifecycle:

1. reserve
2. execute
3. commit actual usage or release the remainder

Each part exists because execution is messy.

### Reserve

Reserve creates bounded room to act before work begins.

### Commit

Commit reconciles estimated usage with actual usage after work completes.

### Release

Release returns any unused reservation when work exits early, is canceled, or consumes less than expected.

Without all three, real failure handling becomes unreliable.

## Retries create two different kinds of problems

Retries are often discussed as one thing, but they actually create two different accounting problems.

### 1. Duplicate intent

The same logical operation may be submitted more than once.

Example:

- the client sends a reservation request
- the server processes it
- the response is lost
- the client retries

If the second request is treated as new, the system may reserve twice.

### 2. Duplicate completion

The same execution may attempt to commit or release more than once.

Example:

- a worker completes a task
- commit is sent
- timeout occurs before acknowledgment
- the worker retries the commit

If commit is not idempotent, the system may count actual usage multiple times.

Both problems are common.  
Both must be handled explicitly.

## Why concurrency changes everything

Concurrency is where many “good enough” budget systems fail.

A single-threaded demo can make almost anything look correct.

Production systems are different.

Multiple requests may:

- reserve simultaneously
- commit simultaneously
- release simultaneously
- affect shared parent scopes
- race at both local and ancestor levels

This becomes even more complex in hierarchical models where one action may consume budget from several scopes at once, such as:

- tenant
- workflow
- run

If these mutations are not handled carefully, concurrency breaks the guarantee that budgets are meant to provide.

That is why Cycles is built around deterministic reservation semantics rather than loose after-the-fact reconciliation.

## Hierarchical budgets make naive logic even less safe

Flat counters are already easy to get wrong.

Hierarchical governance makes the problem more important.

Suppose an action must be valid against:

- tenant budget
- workflow budget
- run budget

A naive system might check these one at a time without a coherent control model.

That can create partial success conditions such as:

- local scope looks valid
- ancestor scope is exhausted
- a concurrent request changes shared state between checks
- a retry replays part of the sequence

Now the system has to answer difficult questions:

- was the action really allowed?
- what should be rolled back?
- which scopes were partially consumed?
- did duplicate handling happen consistently?

This is why budget control in autonomous systems cannot be reduced to “just keep a counter.”

## What Cycles is designed to protect against

Cycles is built for conditions like:

- duplicate reservation attempts
- duplicate commit attempts
- duplicate release attempts
- worker crashes after reserve
- worker crashes after partial execution
- network retries
- concurrent reservation pressure
- hierarchical scope contention
- partial completion with leftover reserved budget

These are the conditions that make simple usage tracking insufficient.

They are also the conditions that determine whether a control layer can be trusted in production.

## Why commit and release must be explicit lifecycle events

A common mistake is to assume that if work starts and finishes normally, accounting is easy.

But real systems often produce incomplete execution paths.

For example:

- work reserves budget but exits before making the expensive call
- work consumes only part of the reserved amount
- work completes but the accounting acknowledgment is delayed
- work is retried by a second worker while the first result is uncertain

If commit and release are not first-class lifecycle events, the system has no clean way to reconcile what actually happened.

That creates either leakage or double counting.

Explicit lifecycle events make these transitions governable.

## Why observability alone is not enough

Some teams try to solve these issues with logging, traces, dashboards, and periodic reconciliation.

Those are valuable tools.

They are not the same as runtime correctness.

Observability can tell you:

- a duplicate happened
- a retry occurred
- usage drift appeared
- a workflow behaved oddly

But it cannot prevent the initial overage or race by itself.

A runtime authority must do more than explain failure after the fact.  
It must remain correct enough under failure to make enforcement meaningful.

## A concrete example

Imagine a workflow step that estimates it needs 100 units.

The system reserves 100 and begins execution.

Then:

- the worker calls a model
- the model call succeeds
- the worker crashes before commit
- the job is retried on another worker

Now the platform must reason about several things:

- was the original reservation already created?
- should the retry create another reservation?
- did actual usage already happen once?
- if the retry commits, is that a duplicate or new consumption?
- if the original reservation is still outstanding, when is the remainder released?

This is not a rare corner case.  
This is exactly the kind of ambiguity production systems create.

A runtime model that ignores these realities becomes financially noisy and operationally untrustworthy.

## Cycles is about bounded execution under uncertainty

One of the key ideas behind Cycles is that enforcement has to survive imperfect information.

At the moment a decision is made, the system may not yet know:

- whether a prior request will be retried
- whether a worker will crash
- whether actual usage will equal the estimate
- whether another concurrent path is about to consume shared budget

That is why Cycles does not rely on a single final usage event.

It uses a lifecycle that can tolerate uncertainty more gracefully:

- reserve bounded room first
- execute work
- reconcile actuals later
- return unused remainder
- remain safe under duplicate and concurrent behavior

## What “real failure modes” means in practice

When we say Cycles is built for real failure modes, we mean it is designed for environments where the following are normal:

- retries are expected
- duplicate delivery happens
- workers fail mid-flight
- state transitions are not perfectly synchronized
- multiple actors compete for shared budget
- long-running workflows can outlive the request that started them

This is the world of production autonomous systems.

A budget system that assumes clean sequential execution may work in a demo and fail in the exact situations where control matters most.

## The design goal

The design goal is not to pretend failure disappears.

The design goal is to make budget governance remain meaningful even when failure occurs.

That means the system should strive to ensure that:

- the same logical action is not charged multiple times by accident
- concurrent actions cannot overrun budget due to naive race conditions
- partial execution can be reconciled explicitly
- reservations do not leak forever
- retries do not make accounting non-deterministic
- enforcement remains understandable under load

This is what separates a production control layer from a reporting wrapper.

## Summary

Autonomous systems operate in an environment shaped by retries, crashes, duplicates, and concurrency.

Any budget control model that ignores these realities will eventually produce drift, ambiguity, or broken enforcement.

That is why Cycles is built around:

- reservation before execution
- commit of actual usage afterward
- release of unused remainder
- idempotent lifecycle handling
- concurrency-aware budget enforcement
- hierarchical policy evaluation across scopes

These are not implementation details.

They are the difference between “tracking usage” and **governing execution under real production conditions**.

## Next steps

To explore the Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)
- [5 Real-World AI Agent Failures That Budget Controls Would Have Prevented](/blog/ai-agent-failures-budget-controls-prevent) — concrete incidents involving retry storms and concurrency races
- [From Observability to Enforcement](/concepts/from-observability-to-enforcement-how-teams-evolve-from-dashboards-to-budget-authority) — how teams evolve from dashboards to runtime authority
- [Cycles vs Custom Token Counters](/concepts/cycles-vs-custom-token-counters) — why ad hoc counters break under the concurrency and retry patterns discussed here



# Runtime Authority vs Runtime Authorization

Two governance terms have started circulating in the AI agent ecosystem, and they sound like the same thing. They aren't.

> **Runtime *authorization*** asks whether an identity is *allowed* to use a tool.
> **Runtime budget authority** asks whether the caller's submitted amount fits the matching ledgers for this next step.

**Authorization grants access; budget authority meters caller-submitted exposure.**

A production agent stack needs both. They sit at different layers, fire at different moments, and bound different things. AWS AgentCore Policy, Akeyless Agentic Runtime Authority, and internal agent-IAM patterns focus on identity, intent, access, and real-time policy enforcement — they decide whether an agent identity, intent, and request context are *permitted* to use a given tool or system. Cycles focuses on bounded exposure: whether a caller-assigned amount can still be reserved against the relevant scoped budget. The caller can express exposure as money, tokens, credits, or `RISK_POINTS`; the current Cycles server does not infer risk or classify tools itself.

The term "runtime authority" is used by multiple vendors with overlapping but different scopes. In Cycles, the concrete decision is narrower: *can this amount be reserved against the configured ledgers now?* The host combines that answer with identity and tool authorization.

## The two questions, side by side

| | Runtime Authorization | Runtime Authority (Cycles) |
|---|---|---|
| **What it answers** | "Is this identity allowed to call this tool?" | "Can this submitted amount be reserved against the matching budgets?" |
| **When it fires** | At identity-resolution time, per tool invocation | At every reservation, before each costly action |
| **What it bounds** | Static policy — which identities can touch which tools | Dynamic budget — total spend or caller-assigned exposure such as credits and risk points |
| **What it does NOT cover** | Cumulative consumption, hierarchical scopes, atomic concurrency | Identity-to-tool mapping, credential management, secret rotation |
| **Decision model** | ALLOW / DENY based on identity and policy | Preflight: ALLOW / [ALLOW_WITH_CAPS](/blog/what-is-runtime-authority-for-ai-agents) / DENY based on scoped budget evaluation; live reserve: ALLOW / ALLOW_WITH_CAPS or an error |
| **State** | Stateless policy lookup (typically) | Persistent budget ledger with [reserve-commit lifecycle](/protocol/how-reserve-commit-works-in-cycles) |

Both layers fire pre-execution. They're complementary — neither makes the other redundant.

## Where each fits in the production stack

A real agent action goes through both layers in sequence:

```text
1. Agent decides to call tool X
2. AUTHORIZATION: "Is this agent identity allowed to invoke X?"
   ↓ Yes (or DENY → caller informed)
3. BUDGET AUTHORITY: "Can the caller-assigned exposure be reserved for this action?"
   ↓ ALLOW or ALLOW_WITH_CAPS (or a budget error → graceful degradation)
4. Execute tool with the constraints from authority's caps
5. Authority commits actual cost, releases unused budget
```

Skip layer 2 and an agent with credentials may reach tools it should not use. Skip layer 3 and an authorized agent has no Cycles ledger bounding cumulative submitted spend or exposure.

## Where adjacent tools fit

We don't ship per-vendor comparison pages against the identity-based agent governance tools — they solve a different problem, and head-to-head framing implies substitution where the right framing is composition. But you should know how Cycles overlaps with what's emerging in this space.

| | Identity / intent-scoped tool access | Per-action risk budget | Pre-execution cost authority | Reserve-commit semantics | Self-hosted, no prompt storage |
|---|:---:|:---:|:---:|:---:|:---:|
| AWS Bedrock AgentCore Policy | Yes | Not publicly documented | Not publicly documented | Not publicly documented | AWS-managed |
| Akeyless Agentic Runtime Authority | Yes — intent-aware access / real-time policy | Not publicly documented | Not publicly documented | Not publicly documented | Cloud / vendor-managed |
| Generic agent IAM patterns | Yes | Usually no | Usually no | No | Varies |
| **Cycles** | API permissions only; downstream tool IAM external | **Caller-assigned RISK_POINTS budget** | **Yes** | **Yes** | **Yes** |

The first column is the authorization / intent-policy layer. AgentCore and Akeyless are well-suited for it — they handle identity, intent-aware access, policy attachment, and credential governance. The middle columns are the bounded-exposure layer — that is where Cycles operates. The final column is a deployment / privacy distinction, not a runtime-authority capability per se.

## Better together

A production stack wires both layers in the order shown above. Cycles supplies API keys with permission scopes (`reservations:create`, `balances:read`, `admin:write`, etc.) for the runtime plane, and identity-based authorization tools handle the upstream question of whether the agent identity is allowed to obtain those keys in the first place.

Concrete example — a SaaS deploying customer-support agents:

- **Authorization layer** (AgentCore / Akeyless / IAM): defines that *the support agent's identity* is allowed to call the `send_email` tool, and *the engineering agent's identity* is allowed to call the `deploy_service` tool. Cross-access denied at the policy layer.
- **Authority layer** (Cycles + host integration): defines that the *support tenant* has $500/month in tokens and a 200-point daily risk budget. The host classifies each email as 40 [RISK_POINTS](/concepts/action-authority-controlling-what-agents-do) and reserves that amount before dispatch. Even though the support agent is *authorized* to send emails, the 6th live reservation fails once that risk budget is exhausted; an LLM call likewise does not proceed when its token reservation fails.

Without authorization, a credentialed caller may reach tools it should not use. Without a cumulative budget or count control, an authorized agent can repeat a tool until some other limit stops it.

## When you only need authorization

- Single-tool agents with low blast radius (read-only, no concurrency, no multi-tenancy).
- Internal-only deployments where the question is "who's allowed to use this tool" and there's no budget to bound.
- Pre-production prototypes where cumulative cost isn't yet a concern.

If you're here, AgentCore Policy or a similar identity-based system is sufficient — Cycles adds overhead you don't need yet.

## When you need authority

- Multi-tenant SaaS where one customer's runaway must not affect other tenants.
- Agents with hierarchical standard scopes—tenant → workspace → app → workflow → agent → toolset—that need multiple budget ledgers. A run can be represented by a unique workflow value when it needs its own ledger.
- Tools with side effects (email, deploy, mutation) where you want to bound risk *separately* from cost.
- Multi-agent delegation chains where authority should attenuate at each hop, not propagate.
- Production cost predictability — you need evidence that configured budgets bound covered execution paths under concurrency and retries.

If any of these apply, identity authorization alone leaves the budget and risk dimensions unbounded. That's where Cycles fits.

## Sources

- [Policy in Amazon Bedrock AgentCore — Control Agent-to-Tool Access](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/policy.html) — AWS documentation on AgentCore Policy enforcement before tool execution.
- [Akeyless launches Runtime Authority for AI Agents](https://www.akeyless.io/press-release/akeyless-launches-runtime-authority-for-ai-agents/) — Akeyless announcement framing identity-aware enforcement as runtime authority.

External vendor capabilities verified against linked sources as of July 2026. These tools evolve quickly — check the linked docs for the latest. Cycles capability statements describe the currently shipped server; protocol-only action-governance features are identified separately in the linked action-authority documentation.

## Related

- [Cycles Protocol](/protocol/) — the open specification behind the runtime-authority claim. Explicit conformance criteria and the reference implementation are public.
- [What Is Runtime Authority for AI Agents](/blog/what-is-runtime-authority-for-ai-agents) — the canonical definition we use throughout Cycles documentation.
- [Action Authority — Controlling What Agents Do](/concepts/action-authority-controlling-what-agents-do) — composing host authorization with caller-assigned RISK_POINTS and host-applied caps.
- [Comparisons — How Cycles Differs from Alternatives](/concepts/comparisons) — proxy/observability/rate-limit comparison hub for the LiteLLM/Helicone/LangSmith axis.
- [Why Rate Limits Are Not Enough](/concepts/why-rate-limits-are-not-enough-for-autonomous-systems) — the deeper argument for why velocity controls and identity policy alone fail for autonomous systems.



# Webhooks and Events

Cycles services emit **events** from implemented budget, reservation, tenant, API-key, webhook, and system hooks. The schema also registers planned event types that are not emitted yet; the [Event Payloads Reference](/protocol/event-payloads-reference) tracks that status. **Webhooks** deliver emitted events to external endpoints via HTTP POST with HMAC-SHA256 signatures.

## Core Concepts

### Events

An event is an immutable record of an emitted decision, lifecycle outcome, or state change. Every event has:
- **event_id** — unique identifier; the key consumers dedupe on
- **event_type** — dotted format like `budget.exhausted` or `reservation.denied`
- **category** — one of: budget, reservation, tenant, api_key, policy, webhook, system
- **timestamp** — when the event occurred
- **tenant_id** — which tenant is affected
- **source** — which service emitted it (an open string; current reference values include `cycles-server`, `cycles-admin`, and `cycles-events`)
- **data** — optional event-specific payload (varies by type)

Events are stored in Redis with a 90-day TTL (configurable).

### Webhook Subscriptions

A subscription defines which events to deliver and where:
- **url** — HTTPS endpoint to receive HTTP POST requests
- **event_types** — specific events to receive (e.g., `["budget.exhausted", "reservation.denied"]`)
- **event_categories** — receive all events in a category (additive with event_types)
- **scope_filter** — optional scope-path filter; only events whose scope matches are delivered (see [Webhook Scope Filter Syntax](/protocol/webhook-scope-filter-syntax))
- **signing_secret** — HMAC-SHA256 key for payload verification

### Delivery Semantics

- **At-least-once** — events may be delivered more than once. Deduplicate using `event_id`.
- **No delivery-order guarantee** — concurrent delivery workers and retries can reorder deliveries. Use `event_id` only for deduplication; reconstruct chronology from event timestamps, the stored event log, and current domain state.
- **Non-blocking** — webhook delivery never blocks the API operation that produced the event.
- **Retry with backoff** — failed deliveries retry with exponential backoff (default: 5 retries).
- **Auto-disable** — subscriptions are disabled after consecutive failures (default: 10).

## Architecture

<EventFlowDiagram />

The events service is **optional**. If not deployed, events accumulate in Redis with TTL and are delivered when the service starts.

## 51 Registered Event Types

| Category | Count | Registered examples (some remain planned) |
|---|---|---|
| budget | 17 | `budget.exhausted`, `budget.threshold_crossed`, `budget.over_limit_entered`, `budget.funded`, `budget.closed_via_tenant_cascade` |
| reservation | 6 | `reservation.denied`, `reservation.commit_overage`, `reservation.released_via_tenant_cascade` |
| tenant | 6 | `tenant.created`, `tenant.suspended`, `tenant.closed` |
| api_key | 7 | `api_key.created`, `api_key.revoked`, `api_key.revoked_via_tenant_cascade` |
| policy | 3 | `policy.created`, `policy.updated`, `policy.deleted` |
| webhook | 7 | `webhook.created`, `webhook.paused`, `webhook.disabled_via_tenant_cascade` |
| system | 5 | `system.store_connection_lost`, `system.webhook_delivery_failed` |

The four `*_via_tenant_cascade` types were added to the enum in governance spec revision v0.1.25.35.

## Tenant Self-Service

Tenants can create their own webhook subscriptions via `/v1/webhooks` (requires `webhooks:write` permission). Tenant webhooks are restricted to budget, reservation, and tenant events: 29 of the 51 registered event types — including the `budget.*` and `reservation.*` `_via_tenant_cascade` fan-out events emitted during a tenant close (see [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics)). API key, policy, webhook lifecycle, and system events are admin-only: a tenant-owned subscription can neither carry them nor receive them from the event stream, enforced at write, dispatch, and last-mile delivery (governance WEBHOOK SUBSCRIPTION INVARIANT 2; the one exception is the owner-triggered `/test` probe — see [Tenant-accessible events](/protocol/webhook-event-delivery-protocol#tenant-accessible-events)).

## Security

- **HMAC-SHA256** — every delivery includes `X-Cycles-Signature: sha256=<hex>` for payload verification
- **Encryption at rest** — signing secrets are encrypted in Redis with AES-256-GCM using `WEBHOOK_SECRET_ENCRYPTION_KEY`; admin and events fail startup without the key unless plaintext is explicitly enabled for local development
- **SSRF prevention** — private IP ranges blocked by default, HTTPS required in production

## Learn More

- [Webhook Integrations Guide](/how-to/webhook-integrations) — PagerDuty, Slack, ServiceNow examples with code
- [Security Hardening](/how-to/security-hardening) — webhook URL security and secret rotation
- [Production Operations](/how-to/production-operations-guide) — events service deployment and failure handling



# What Cycles Is Not: Billing, Rate Limiting, Orchestration, and Other Category Confusion

When people first encounter Cycles, they often try to map it onto an existing category.

That is normal.

Most infrastructure projects are easier to understand when they resemble something familiar. So the first questions are often:

- Is this just billing?
- Is this a rate limiter?
- Is this workflow orchestration?
- Is this observability for AI usage?
- Is this a policy engine?
- Is this a gateway or proxy?
- Is this an AI safety product?

The honest answer is:

**Cycles overlaps with some of these categories, but it is not reducible to any of them.**

Cycles is a **runtime authority for autonomous agents**.

That means it exists to decide whether autonomous work is allowed to proceed, how much bounded exposure it may reserve, and how that usage is reconciled afterward.

This article explains what Cycles is **not**, and why those distinctions matter.

## Cycles is not billing

Billing tells you what to charge.

Cycles tells you what may execute.

Billing usually answers questions like:

- how much did this customer use?
- what invoice should we generate?
- how should usage be priced?
- how should this appear in finance or revenue systems?

Cycles is not designed to replace that.

Cycles operates earlier in the chain.

It answers questions like:

- may this action proceed right now?
- how much exposure can it reserve before execution?
- what scope should this count against?
- what happens if the budget is exhausted?
- what should be committed as actual usage after the action completes?

Billing is usually retrospective.

Cycles is pre-execution and execution-aware.

The two can work together, but they are not the same thing.

## Cycles is not rate limiting

Rate limiting controls **velocity**.

Cycles controls **total bounded exposure**.

A rate limiter might say:

- 100 requests per minute
- 10 tool invocations per second
- 1,000 API calls per hour

That is useful for abuse prevention, fairness, and traffic shaping.

But it does not answer:

- how much total budget may this run consume?
- may this tenant continue if it already exhausted its daily budget?
- should this workflow continue after repeated retries?
- should this tool call proceed if the run is almost out of budget?

An agent can stay perfectly inside its request-per-second threshold and still burn through budget over time.

That is why rate limiting and Cycles solve different problems.

You should usually keep your rate limiter.

Just do not confuse it with budget enforcement.

## Cycles is not observability

Observability tells you what happened.

Cycles helps determine what is allowed to happen.

Observability tools are essential. They help teams answer questions like:

- which workflows are expensive?
- where did retries occur?
- what was the cost distribution?
- which tenant used the most resources?
- what failed, when, and why?

Cycles benefits from good observability.  
It does not replace it.

But observability alone does not create control.

A dashboard may tell you that a runaway workflow consumed too much budget.

Cycles is about introducing a control point **before and during execution**, so that work can be bounded instead of merely explained later.

That is the difference between reporting and governance.

## Cycles is not orchestration

Workflow orchestration decides **what should happen next**.

Cycles decides **whether bounded execution is allowed to continue**.

An orchestrator might manage:

- task sequencing
- retries
- step dependencies
- state transitions
- fan-out and fan-in
- compensation logic

Cycles does not replace that.

It is not trying to become the workflow engine.

Instead, Cycles sits alongside execution and asks:

- can this next step reserve enough budget?
- should this tool path be allowed?
- should this run continue?
- should the system degrade instead of proceed normally?

That makes Cycles complementary to orchestration, not a substitute for it.

## Cycles is not a generic policy engine

Policy engines evaluate rules.

Cycles enforces budget-aware execution semantics.

A generic policy engine may be able to express conditions like:

- user role is admin
- environment is production
- action type is write
- tenant plan is premium

Cycles can certainly work with policy logic.

But its core purpose is narrower and more operational.

It is concerned with things like:

- reservation
- commit
- release
- balances
- bounded execution
- hierarchical budget scopes
- retry-safe lifecycle semantics

In other words, Cycles is not just “if this then allow/deny.”

It is a control model for autonomous work that consumes budgeted exposure over time.

## Cycles is not merely a gateway or proxy

A gateway can be one deployment surface for Cycles.

It is not the category itself.

You could embed Cycles behind:

- an LLM gateway
- an API proxy
- a service mesh boundary
- a workflow runtime
- an application SDK

Those are all valid places to integrate.

But Cycles is not simply “a proxy that counts requests.”

Its control model is richer than request forwarding.

It needs to support:

- reserve before execution
- commit actual usage afterward (auto-releasing unused remainder)
- release explicitly on cancellation
- hierarchical scope enforcement
- retry-safe and idempotent behavior

A proxy may carry these semantics.  
But the semantics are the important part.

## Cycles is not AI safety in the broad philosophical sense

Cycles can reduce certain classes of operational risk.

It is not a general solution to AI safety.

Cycles does not claim to solve:

- hallucinations
- alignment
- truthfulness
- harmful content generation
- model bias
- broad social safety concerns

Its scope is narrower.

Cycles helps govern **cost, side effects, and bounded execution** in autonomous systems.

That includes questions like:

- how much can this system spend?
- what irreversible actions may it take?
- what should happen when the budget is exhausted?
- how do we keep retries and loops from becoming unbounded incidents?

That is valuable.  
It is also specific.

Keeping that boundary clear makes the project more credible, not less.

## Cycles is not just cost tracking

Cost tracking answers:

- how much did we spend?

Cycles answers:

- how much are we willing to let this execution risk before it proceeds?

That distinction is subtle but important.

Tracking is passive.  
Governance is active.

A post-hoc cost report may help you improve later.

A budget authority can stop a run before it becomes a larger incident.

## Cycles is not the same as quotas

Quotas are static boundaries.

Cycles provides a runtime lifecycle.

A quota might say:

- this tenant gets 10,000 units per day
- this workflow gets 500 units per run

That is useful.

But autonomous systems also need a way to manage execution as it happens:

- reserve estimated exposure
- execute work
- commit actual usage (auto-releases unused remainder)
- release explicitly on cancellation
- handle retries safely
- reconcile partial completion

That is the part Cycles focuses on.

Quotas are part of policy.  
Cycles is the runtime discipline that makes policy operational.

## Cycles is not a token or rewards scheme

The name can sometimes lead people in the wrong direction.

Cycles is not a speculative asset, loyalty point, or incentive token.

It is an accounting and governance primitive.

A Cycle is an operator-defined unit of bounded exposure.

That unit may represent cost, side-effect potential, or some normalized execution budget. But its role is operational, not financialized.

The point is to make autonomous systems governable, not tradable.

## So what is Cycles, exactly?

The shortest answer is:

**Cycles is a runtime authority for autonomous agents.**

More specifically, it is a protocol and runtime model for:

- reserving bounded exposure before work starts
- committing actual usage after work completes
- releasing unused remainder
- enforcing limits across scopes such as tenant, workflow, and run
- remaining meaningful under retries, duplicates, crashes, and concurrency

That makes it adjacent to several existing categories, but not identical to any one of them.

## Why this distinction matters

Category confusion is not just a messaging problem.

It leads to the wrong adoption expectations.

If someone thinks Cycles is billing, they will ask where invoices are.

If someone thinks it is rate limiting, they will judge it against request throttling.

If someone thinks it is orchestration, they will expect workflow graphs and schedulers.

If someone thinks it is observability, they will look for dashboards first.

Those are all reasonable expectations for those categories.

They are the wrong expectations for Cycles.

The right expectation is:

**Cycles helps teams bound and govern autonomous execution before it becomes unbounded cost or side effect.**

That is the job.

## When Cycles is the right fit

Cycles is a good fit when a system needs more than reporting or traffic shaping.

For example:

- long-running agent loops
- tool-calling workflows
- multi-tenant AI platforms
- background autonomous jobs
- systems where retries and fan-out affect cost materially
- systems where side effects need budget-aware control
- systems that need pre-execution budget checks, not just dashboards

In these cases, the gap is usually not visibility.

It is the absence of a runtime authority.

## Summary

Cycles is related to billing, rate limiting, observability, orchestration, and policy.

But it is not any one of those things.

It is its own control layer.

That control layer exists to make autonomous execution:

- explicit
- bounded
- budget-aware
- retry-safe
- enforceable across scopes

That is why the best way to think about Cycles is not as a dashboard, a proxy, or a scheduler.

It is a **runtime authority for autonomous agents**.

## Next steps

To explore the Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)
- [AI Agent Cost Management: The Complete Guide](/blog/ai-agent-cost-management-guide) — the maturity model from monitoring to hard enforcement, and why Cycles is specifically Tier 4
- [Cycles vs Rate Limiting](/concepts/cycles-vs-rate-limiting) — the detailed comparison between rate limiting and runtime authority
- [Runtime Authority vs Runtime Authorization](/concepts/runtime-authority-vs-runtime-authorization) — Cycles is also not runtime authorization. AWS Bedrock AgentCore Policy and Akeyless answer "is this identity allowed"; Cycles answers "does this agent still have bounded permission to take this next step".
- [Comparisons](/concepts/comparisons) — how Cycles differs from rate limiters, provider caps, observability tools, and more



# Why Coding Agents Do Not Replace Cycles

The real risk with coding agents is not that they fail to produce work.

It is that they make production cheap enough to blur the line between **more output** and **more value**.

When implementation gets dramatically easier, teams do not automatically become more disciplined. In many cases, they become less so. Scope expands. Nice-to-haves slip in. Work that would once have been deferred now feels cheap enough to keep going.

That is where Cycles matter.

This article is about the **business-layer governance problem**: how teams decide what work is worth funding, where scope should stop, and when priorities should be re-evaluated.

The runtime-layer problem is different. It asks how an autonomous system is prevented from exceeding its allowed execution budget in the middle of a run — under retries, parallelism, partial failure, and tool fan-out. That is covered separately in [Coding Agents Need Runtime Authority](/concepts/coding-agents-need-runtime-budget-authority).

The two are complementary, not interchangeable.

Coding agents increase execution capacity.  
Runtime authority bounds execution.  
Cycles govern whether the work was worth funding in the first place, and whether the next unit of work deserves more budget.

## Agents optimize for output. Cycles optimize for value.

A coding agent's job is to complete the task it was given.

Give it a prompt and it will try to produce output: code, tests, documentation, refactors, fixes, follow-up patches. That is what makes it useful.

But output and value are not the same thing.

A team can now generate more implementation work than ever before with less friction than ever before. The bottleneck is no longer only execution. Increasingly, the bottleneck is deciding what is actually worth continuing, polishing, expanding, or funding.

That is where Cycles matter.

A Cycle creates a forcing function around value, not just activity. It establishes a bounded unit of committed work and a deliberate checkpoint afterward: was the outcome worth the spend, did the scope remain justified, and what should receive budget next?

A coding agent can help produce more output within that boundary. It cannot create the boundary, own the tradeoff, or decide whether the result justified further investment.

## The hidden cost of cheap output

When output becomes cheaper, scope tends to expand.

That is not because teams become irrational. It is because each incremental addition starts to feel inexpensive in isolation.

Add the extra edge case.  
Support one more path.  
Polish the interface.  
Refactor the surrounding module.  
Generate another round of tests.  
Handle one more environment.

Each decision can sound reasonable on its own. The problem is cumulative. When the friction of implementation falls, the friction that used to enforce prioritization disappears with it.

That is how teams drift from a small, valuable deliverable into a much larger body of work that no one explicitly decided was worth funding.

Cycles restore that missing discipline.

They provide a business boundary: this is what we are funding now, this is what counts as done for this period, and this is the checkpoint before more scope is authorized.

That is a different problem from runtime enforcement inside a single autonomous run. If the question is what happens when an agent retries, fans out, or exceeds its allowed execution budget mid-run, see [Coding Agents Need Runtime Authority](/concepts/coding-agents-need-runtime-budget-authority). If the question is whether the team should continue funding the next increment of work at all, that is the role Cycles play.

## Spend visibility is not value accountability

Agents make usage easier to measure.

You can inspect token consumption, tool calls, session traces, CI minutes, and provider bills with much greater precision than before. That visibility is useful. It helps explain what happened.

But it still does not tell you whether the output was worth the cost.

That judgment does not emerge automatically from logs, traces, or model invoices. It remains a human governance decision.

Someone still has to decide:

Was this worth the spend?  
Did this work move the priority that mattered?  
Should this area receive more budget, or should the team stop here?  
Did the agent help compress valuable work, or did it simply make it easier to produce more of it?

Cycles exist to force that conversation at a predictable boundary.

Without a structure like that, teams can become highly efficient at producing output while becoming much less disciplined about deciding whether that output deserved to exist.

## Cycles get cheaper. They do not go away.

The right reframe is simple:

::: info
Coding agents do not eliminate Cycles. They reduce the cost of executing within them.
:::

If a team previously needed six engineers to hit a given scope and can now hit the same scope with four plus coding agents, that is a real productivity gain. But the Cycle still matters. The checkpoint still matters. The prioritization still matters. The budget decision still matters.

What changed is not the need for governance.

What changed is the cost curve inside the governance structure.

That is where the real upside of coding agents shows up. Teams that understand this use automation to make each governed iteration faster and cheaper. Teams that ignore it often confuse increased output with increased progress, then discover later that they have accumulated a large body of work with weak linkage to business outcomes.

## Runtime control and business control are not the same

It is worth stating the distinction directly.

Runtime authority answers questions like:

- Can this next autonomous step proceed?
- Should this run be denied, degraded, or stopped?
- What happens under retries, concurrency, and partial failure?
- How is budget reserved, committed, and released during execution?

That is an execution-layer control problem.

Cycles answer a different set of questions:

- Was this slice of work worth funding?
- Should we keep investing here?
- Did the delivered output justify the committed spend?
- What deserves budget next?

That is a business-layer governance problem.

Both matter.

If you have runtime control without Cycles, you may prevent overruns inside execution while still funding the wrong work.  
If you have Cycles without runtime control, you may make good planning decisions while still allowing autonomous runs to exceed safe limits in practice.

The systems complement each other because they constrain different failure modes.

## The more interesting future: budgeting outcomes, not just features

As teams get better at using coding agents, the natural next step is not simply to ship more tickets.

It is to become more explicit about the outcome being funded.

Instead of thinking only in terms of budget per feature, more mature teams will increasingly think in terms of budget per outcome: move this metric, improve this workflow, reduce this latency, increase this conversion, lower this support burden.

That is where Cycles become even more useful.

A Cycle boundary is a natural point to ask not just whether the implementation was completed, but whether the work moved the thing that mattered. As coding agents make delivery cheaper, outcome discipline becomes more important, not less. Otherwise teams risk becoming extremely efficient at completing tasks that should not have received additional budget.

## Bottom line

Coding agents are a force multiplier on execution.

Runtime authority ensures autonomous execution stays bounded while it is happening.

Cycles provide the governance structure above that layer: the discipline that asks whether the work was worth funding, whether scope should stop, and what should receive budget next.

You need all three ideas if you want agentic software to be both fast and economically coherent.

If you want the runtime-side companion to this piece — reservations, enforcement, retries, concurrency, and bounded execution inside a single agent run — see [Coding Agents Need Runtime Authority](/concepts/coding-agents-need-runtime-budget-authority).

The teams that get the most out of coding agents will not be the ones that simply generate the most output.

They will be the ones that pair machine-speed execution with explicit runtime control and deliberate budget judgment.

## Next steps

To learn more:

- Read [Coding Agents Need Runtime Authority](/concepts/coding-agents-need-runtime-budget-authority) for the runtime-layer companion to this piece
- Understand [Why Rate Limits Are Not Enough](/concepts/why-rate-limits-are-not-enough-for-autonomous-systems) for how velocity controls differ from runtime authority
- See [From Observability to Enforcement](/concepts/from-observability-to-enforcement-how-teams-evolve-from-dashboards-to-budget-authority) for how teams evolve from dashboards to budget governance
- Explore the [reserve-commit lifecycle](/protocol/how-reserve-commit-works-in-cycles) that powers runtime enforcement
- Get started with the [Python Client](/quickstart/getting-started-with-the-python-client) or [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- [AI Agent Budget Control: Enforce Hard Spend Limits](/blog/ai-agent-budget-control-enforce-hard-spend-limits) — the technical mechanism behind runtime budget enforcement



# Why Rate Limits Are Not Enough for Autonomous Systems

Autonomous systems do not fail like traditional software.

They do not simply receive a request, process it once, and return a response.

They loop.  
They retry.  
They fan out across tools and models.  
They continue after partial failure.  
They make decisions that create cost and side effects over time.

That changes the control problem.

For traditional APIs, controls like rate limits, quotas, and timeouts are often enough. They help bound request velocity and reduce abuse.

For autonomous systems, they are not enough.

## The real problem is not speed

Rate limits answer a narrow question:

**How fast can this system act?**

Autonomous systems introduce a different question:

**How much total exposure is this system allowed to create?**

That exposure may include:

- LLM usage and token spend
- external API calls
- database writes
- message dispatch
- payment instructions
- workflow fan-out
- tool invocations with irreversible side effects

A system can remain within its request-per-second threshold and still create unacceptable cost or damage over time.

That is why teams often discover the problem too late.

Not when the first request succeeds.  
Not when latency rises.  
But when the bill arrives, a workflow loops indefinitely, a tool runs recursively, or one tenant quietly consumes more than intended.

## Why rate limits fail in practice

Rate limits are useful. They should not be removed.

They are just solving a different problem.

Here are some common failure cases where rate limits are insufficient.

### 1. Loops stay within allowed velocity

An agent can call a model every few seconds, remain fully within rate limits, and still burn through budget over hours.

Nothing is “spiking.”  
Nothing looks like abuse.  
The system is simply allowed to continue.

### 2. Retries multiply total cost

A failed step retries.  
Then retries again.  
Then downstream steps retry too.

Each individual request may be valid.  
The accumulated exposure is not.

### 3. Tool calls create hidden fan-out

A single high-level action can expand into:

- multiple model calls
- several external APIs
- database writes
- follow-up jobs
- additional agent steps

Rate limits see individual calls.  
They do not naturally bound the full execution chain.

### 4. Per-request controls ignore tenant-level consumption

A multi-tenant platform may cap each request correctly but still fail to enforce what one tenant is allowed to consume over a run, workflow, or billing window.

### 5. Post-hoc observability is not enforcement

Dashboards can show what happened after the fact.

That is useful for analysis.  
It is not the same as deciding, before execution, whether an action is allowed to proceed.

## Autonomous systems need runtime authority

The missing primitive is not better logging.

It is not another dashboard.

It is a way to make autonomous work ask for bounded room to act **before** it acts.

That requires a different control model:

1. declare intent
2. reserve budget
3. execute
4. commit actual usage or release the remainder

This is the model behind Cycles.

## The Cycles model

Cycles introduces deterministic budget control for autonomous execution.

Instead of discovering cost and side effects only after they occur, a system reserves bounded exposure before work begins.

At a high level:

- an action declares expected usage
- budget is reserved against one or more scopes
- work executes only if reservation succeeds
- actual usage is committed afterward (unused remainder is released automatically)
- or the reservation is released explicitly if work is canceled

This changes the control surface from:

::: info
“observe what happened”
:::

to:

::: info
“authorize bounded execution, then reconcile actual usage”
:::

That difference matters under retries, crashes, concurrency, and long-running workflows.

## Why reserve and commit are different from simple quotas

A quota says:

::: info
you may use up to this much over time
:::

A reserve-commit model says:

::: info
this execution is allowed to consume up to this bounded amount now
:::

That makes several important things possible.

### Bounded execution before work starts

If the system cannot reserve enough budget, the action can be denied, degraded, or rerouted before cost is incurred.

### Safer retries

If retries are idempotent and tied to the same reservation lifecycle, the system can avoid accidental double-spend.

### Actuals instead of guesswork

Many systems can estimate cost before execution but only know the true cost afterward.

Reserve-commit handles both.

### Hierarchical control

A single action may need to satisfy limits at multiple levels:

- tenant
- workspace
- app
- workflow
- agent
- toolset

This is hard to model cleanly with flat quotas alone.

## A concrete example

Imagine a customer support agent that can:

- call an LLM
- query a CRM
- search a knowledge base
- send an email
- open a ticket
- trigger a follow-up workflow

A rate limiter can throttle each component.

But it does not answer:

- how much total budget is this run allowed to consume?
- how much can this tenant spend today?
- should this workflow continue if prior retries already consumed most of its budget?
- should the system downgrade from a larger model to a smaller one?
- should the email or ticket-creation step be blocked once the run is over budget?

Those are runtime authority questions, not velocity questions.

## What teams usually do today

Most teams solve this in ad hoc ways:

- model-specific caps
- provider dashboards
- cron-based alerts
- tenant usage counters
- best-effort checks inside business logic
- manual kill switches
- custom retry heuristics

These can help, but they are often fragmented and hard to make correct under concurrency.

The result is usually one of two extremes:

- controls are too weak and failures become expensive
- controls are too rigid and autonomous systems become brittle

A proper runtime authority gives teams a cleaner middle ground.

## What Cycles is for

Cycles is designed for teams building systems where autonomous software can create real cost or irreversible side effects.

That includes:

- agent loops
- tool-calling workflows
- long-running background execution
- multi-tenant AI platforms
- budget-sensitive inference systems
- infrastructure that must hold up under retries and partial failure

Cycles is not a billing dashboard.  
It is not just observability.  
It is not a rate limiter.

It is a control layer for bounded autonomous execution.

## The practical takeaway

Keep your rate limits.

But do not confuse them with budget enforcement.

Rate limits are good at controlling speed.  
Autonomous systems also need controls for total exposure.

That means introducing a system that can:

- reserve budget before work starts
- commit actual usage afterward (auto-releasing unused remainder)
- release explicitly on cancellation
- apply limits across scopes
- remain safe under retries and concurrency

That is the problem Cycles exists to solve.

## Next steps

To learn more:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage tenants and budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)
- [The True Cost of Uncontrolled AI Agents](/blog/true-cost-of-uncontrolled-agents) — real-world failure modes and costs of running agents without budget limits

---

# Protocol Reference



# Action Governance Preview in Cycles

::: warning Preview status
The v0.1.26 action-governance specs are upcoming extension specs. The active v0.1.25 conformance target is still the required implementation surface, and the current reference servers do not enforce v0.1.26 action quotas or action-kind allow/deny lists at runtime.

Some v0.1.25.x admin endpoints accept forward-compatible query parameters described below, but those parameters are compatibility hooks until a server implements the v0.1.26 extension semantics.
:::

Action governance adds a second authority layer alongside spend budgets: a server can meter and block by **what kind of action** an agent is about to execute, not only by how much budget the action consumes.

Budgets still answer "can this tenant spend more?" Action governance answers questions such as:

- Can this agent send email at all?
- Has this run already used its five `message.email.send` calls?
- Are high-risk actions limited more tightly than read-only actions?
- Can this policy be observed in shadow mode before it starts blocking production traffic?

## Spec Files

The preview is split across three YAML specs in [`runcycles/cycles-protocol`](https://github.com/runcycles/cycles-protocol):

| Spec | Owns |
|---|---|
| [`cycles-action-kinds-v0.1.26.yaml`](https://github.com/runcycles/cycles-protocol/blob/main/cycles-action-kinds-v0.1.26.yaml) | Canonical action-kind registry, risk classes, quota windows, action quota schemas, `GET /v1/action-kinds`, `GET /v1/action-kinds/{kind}`, `GET /v1/admin/action-quota-counters`, and `POST /v1/admin/action-quota-counters/reset` |
| [`cycles-protocol-extensions-v0.1.26.yaml`](https://github.com/runcycles/cycles-protocol/blob/main/cycles-protocol-extensions-v0.1.26.yaml) | Runtime behavior: `DenyDetail`, `ObserveModeEnum`, action-governance reason codes, observed events, quota events, and full reservation evaluation order |
| [`cycles-governance-extensions-v0.1.26.yaml`](https://github.com/runcycles/cycles-protocol/blob/main/cycles-governance-extensions-v0.1.26.yaml) | Admin-plane extensions: `Policy.action_quotas`, `Policy.risk_class_quotas`, `Policy.allowed_action_kinds`, `Policy.denied_action_kinds`, `Tenant.observe_mode`, overview aggregates, and preview list filters |

The YAML files remain the authority. This page is the human-readable map.

## Action Kinds

An action kind is a stable string for the operation an agent is about to perform, such as `llm.completion`, `web.search`, `message.email.send`, or `code.exec`.

Each kind belongs to an `ActionRiskClass`:

| Risk class | Meaning |
|---|---|
| `read_only` | Reads state without mutation |
| `local_mutation` | Mutates local or reversible state |
| `side_effect` | Produces effects that can drive downstream work |
| `external_side_effect` | Mutates an external system |
| `high_risk` | High-blast-radius action that should usually be explicitly allowed |

Servers implementing the action-kind registry expose:

```bash
curl -G "http://localhost:7878/v1/action-kinds" \
  --data-urlencode "risk_class=external_side_effect" \
  --data-urlencode "deprecated=false" \
  --data-urlencode "limit=50"
```

`GET /v1/action-kinds` is public in the preview spec and supports `risk_class`, `deprecated`, `cursor`, and `limit`. `GET /v1/action-kinds/{kind}` performs an exact lookup:

```bash
curl "http://localhost:7878/v1/action-kinds/message.email.send"
```

## Policy Fields

The governance extension adds four policy fields:

| Field | Purpose |
|---|---|
| `action_quotas` | Per-action count quota rules, such as five email sends per run |
| `risk_class_quotas` | Aggregate count quota rules by risk class, such as two `high_risk` actions per day |
| `allowed_action_kinds` | Allowlist; when non-empty, only listed kinds may pass |
| `denied_action_kinds` | Denylist; listed kinds are blocked before quota and budget checks |

`allowed_action_kinds` and `denied_action_kinds` are mutually exclusive when both are non-empty. A server implementing v0.1.26 must reject that policy shape instead of guessing precedence.

Tenant records also gain `observe_mode` with values `ENFORCE`, `OBSERVE`, and `DISABLED`. `ENFORCE` is normal behavior. `OBSERVE` evaluates the rules and emits observed events without mutating balances or quota counters. `DISABLED` is a synonym for `ENFORCE` in the extension spec.

## Evaluation Order

For a reservation request, v0.1.26 servers evaluate action governance before budget reservation:

1. Action-kind access control from `denied_action_kinds` and `allowed_action_kinds`.
2. Risk-class quotas from `risk_class_quotas`.
3. Per-kind quotas from `action_quotas`.
4. Budget checks and caps from the existing Cycles reserve/decide model.
5. Atomic reservation and counter mutation when the request is allowed and not in observe/dry-run mode.

That order matters. A denied action kind never consumes quota and never reaches the budget check. In `OBSERVE` mode, the server performs the same evaluation but does not mutate balances or quota counters.

## Quota Windows

Action quotas use `ActionQuotaWindow`:

| Window | Use |
|---|---|
| `per_run` | Tied to `Subject.dimensions.run_id`; requires a run id |
| `per_minute_tumbling` | UTC-minute bucket for burst control |
| `per_hour_tumbling` | UTC-hour bucket |
| `per_day_tumbling` | UTC-day bucket |
| `per_tenant_per_day` | Tenant-root daily bucket, independent of nested subject scope |

Per-kind quotas and risk-class quotas use the same window model. A single allowed reservation may increment both a per-kind counter and a risk-class counter when both rules match.

## Counter APIs

Servers implementing the preview expose an operator view of live counters:

```bash
curl -G "http://localhost:7979/v1/admin/action-quota-counters" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  --data-urlencode "tenant_id=acme-corp" \
  --data-urlencode "scope=tenant:acme-corp" \
  --data-urlencode "action_kind=message.email.send" \
  --data-urlencode "window=per_run" \
  --data-urlencode "run_id=run_123" \
  --data-urlencode "include_zero=false"
```

`GET /v1/admin/action-quota-counters` is a debug/read endpoint for current counter state. The preview adds an `action_quotas:read` permission; v0.1.26 also allows `balances:read` as a deprecated fallback, with the fallback removed in v0.1.27.

Counter reset is intentionally narrow:

```bash
curl -X POST "http://localhost:7979/v1/admin/action-quota-counters/reset" \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -d '{
    "tenant_id": "acme-corp",
    "scope": "tenant:acme-corp/workspace:prod/agent:support-bot",
    "action_kind": "message.email.send",
    "window": "per_run",
    "window_key": "run_123",
    "reason": "incident response: undo double-counted email send"
  }'
```

`POST /v1/admin/action-quota-counters/reset` is AdminKeyAuth-only in v0.1.26. It resets exactly one counter identified by `tenant_id`, `scope`, `action_kind`, `window`, and `window_key`; bulk reset is not part of the preview. Risk-class counters use `action_kind="__risk__"` and include `risk_class`. Every successful reset emits a `quota.counter_reset` event.

## Deny And Event Surface

The runtime extension adds `DenyDetail` to denied decisions and conflict errors. For action governance, known reason codes include:

| Reason code | Meaning |
|---|---|
| `ACTION_KIND_DENIED` | The matching policy's denylist blocked the kind |
| `ACTION_KIND_NOT_ALLOWED` | The matching policy's allowlist did not include the kind |
| `ACTION_QUOTA_EXCEEDED` | A per-kind or risk-class quota was exhausted |

When `reason_code=ACTION_QUOTA_EXCEEDED`, `deny_detail.quota_violation` carries structured quota context such as action kind, window, used count, limit, scope, and policy id.

The preview also defines:

- `reservation.observed_denied` for `OBSERVE` evaluations that would deny.
- `reservation.observed_allowed` for `OBSERVE` evaluations that would allow.
- `quota.threshold_approaching` when a quota counter crosses a configured threshold.
- `quota.counter_reset` when an admin resets a counter through the reset API.

## Admin Compatibility Fields

The governance extension reserves several dashboard and list fields. Their semantics are:

| Surface | Field or filter | v0.1.26 meaning |
|---|---|---|
| `GET /v1/admin/overview` | `recent_denials_by_reason` | Count recent denials by reason code, including action-governance reason codes when implemented |
| `GET /v1/admin/overview` | `quota_health` | Summarize action quota counters near or at their limit |
| `GET /v1/admin/overview` | `access_control_stats` | Count policies using allow/deny action-kind lists and related denial activity |
| `GET /v1/admin/overview` | `tenant_counts.in_observe_mode` | Count tenants with `observe_mode != ENFORCE` |
| `GET /v1/admin/tenants` | `observe_mode` | Filter tenants by `DISABLED`, `OBSERVE`, or `ENFORCE` |
| `GET /v1/admin/policies` | `has_action_quotas` | Filter policies with non-empty `action_quotas` or `risk_class_quotas` |
| `GET /v1/admin/policies` | `references_action_kind` | Filter policies whose `action_quotas`, `allowed_action_kinds`, or `denied_action_kinds` mention the given kind; `risk_class_quotas` entries do not match, since they name risk classes rather than kinds |

On v0.1.25.x reference admin servers, `observe_mode`, `has_action_quotas`, and `references_action_kind` are accepted as forward-compatible query parameters but do not narrow results until the v0.1.26 action-governance extension is implemented.

## Implementation Checklist

- Keep the v0.1.25 conformance target green before adding preview behavior.
- Publish `GET /v1/action-kinds` and `GET /v1/action-kinds/{kind}` before allowing policies to reference custom kinds.
- Reject unknown `custom.*` kinds unless the server has an explicit declaration for them.
- Evaluate access control before quotas, and quotas before budget checks.
- Emit observed events in `OBSERVE` mode without mutating balances or quota counters.
- Treat reason-code strings as open values in dashboards and SDKs so future extensions do not break clients.

## Next Steps

- [Protocol Overview](/protocol/) - current conformance target and spec file map.
- [Dry Run and Shadow Mode](/protocol/dry-run-shadow-mode-evaluation-in-cycles) - existing dry-run behavior that observe mode builds on.
- [Error Codes and Error Handling](/protocol/error-codes-and-error-handling-in-cycles) - base and extension denial semantics.
- [Using the Cycles Dashboard](/how-to/using-the-cycles-dashboard) - how overview fields surface in operator workflows.



# API Reference for the Cycles Protocol

This is a developer-friendly reference for the core runtime budget endpoints and public CyclesEvidence retrieval endpoints. The [interactive API reference](/api/) is generated from the YAML spec and remains the exhaustive operation browser.

Tenant-scoped runtime requests require the `X-Cycles-API-Key` header for authentication. Reservation list/detail/release also accept `X-Admin-API-Key` for operator workflows on the small admin-on-behalf-of surface. The two CyclesEvidence read endpoints, `GET /v1/evidence/{evidence_id}` and `GET /v1/.well-known/cycles-jwks.json`, are public by spec because they expose only content-addressed evidence envelopes and public verification keys.

::: info Protocol conformance
Cycles is an **open protocol with a minimum conformance surface**. The active v0.1.25 target requires 12 MUST operations: four core runtime reservation operations plus eight cross-plane event, webhook, balance, and auth-introspection operations. `decide`, reservation listing/detail, and direct-debit events are SHOULD-level runtime operations that the reference servers expose. v0.1.26 action-governance specs are published as upcoming extensions, but they are not required for current conformance and are not enforced by the current reference servers. See [`CONFORMANCE.md`](https://github.com/runcycles/cycles-protocol/blob/main/CONFORMANCE.md) for the authoritative MUST / SHOULD / MAY statement.
:::

## Common headers

### Request headers

| Header | Required | Description |
|---|---|---|
| `Content-Type` | Yes (POST) | `application/json` |
| `X-Cycles-API-Key` | Yes for tenant-scoped runtime endpoints | API key for authentication and tenant derivation |
| `X-Admin-API-Key` | Operator-only on reservation list/detail/release | Admin-on-behalf-of authentication for incident response and inspection |
| `X-Idempotency-Key` | No | Client-provided idempotency key (also accepted in the request body) |

### Response headers

| Header | Description |
|---|---|
| `X-Request-Id` | Unique request identifier for debugging and support |
| `X-Cycles-Trace-Id` | 32-hex W3C Trace Context identifier. Servers MUST echo it on every response (2xx, 4xx, 5xx). The trace ID is taken from an inbound `traceparent` header (preferred) or `X-Cycles-Trace-Id` header when valid, otherwise generated fresh. See [Correlation and Tracing](/protocol/correlation-and-tracing-in-cycles). |
| `X-Cycles-Tenant` | Effective tenant identifier derived from auth context (optional in v0) |
| `X-RateLimit-Remaining` | Number of requests remaining in current window (optional in v0) |
| `X-RateLimit-Reset` | Unix timestamp (seconds) when rate limit resets (optional in v0) |

## Common types

### Subject

The budgeting scope. At least one standard field is required.

```json
{
  "tenant": "acme",
  "workspace": "production",
  "app": "support-bot",
  "workflow": "refund-flow",
  "agent": "planner",
  "toolset": "search-tools",
  "dimensions": {
    "cost_center": "engineering",
    "run_id": "run-12345"
  }
}
```

All fields are optional except that at least one of `tenant`, `workspace`, `app`, `workflow`, `agent`, or `toolset` must be present. The `dimensions` field allows arbitrary key-value pairs for alternative taxonomies — attribution, reporting, and policy facets. Dimensions never derive budget scopes, and v0 servers MAY ignore them for budgeting decisions; anything that needs an enforceable budget belongs in one of the six standard fields.

### Amount

```json
{
  "amount": 5000,
  "unit": "USD_MICROCENTS"
}
```

Units: `USD_MICROCENTS`, `TOKENS`, `CREDITS`, `RISK_POINTS`.

### Action

```json
{
  "kind": "llm.completion",
  "name": "openai:gpt-4o",
  "tags": ["customer-facing", "prod"]
}
```

### Caps (soft constraints)

Returned when the decision is `ALLOW_WITH_CAPS`:

```json
{
  "max_tokens": 500,
  "max_steps_remaining": 3,
  "tool_allowlist": ["search"],
  "tool_denylist": ["code_exec"],
  "cooldown_ms": 2000
}
```

### Error response

```json
{
  "error": "BUDGET_EXCEEDED",
  "message": "Insufficient budget in scope tenant:acme",
  "request_id": "req-abc-123",
  "trace_id": "0af7651916cd43dd8448eb211c80319c",
  "details": {}
}
```

---

## POST /v1/reservations

Reserve budget before executing work.

### Request body

| Field | Type | Required | Description |
|---|---|---|---|
| `idempotency_key` | string | Yes | Unique key for idempotent retries |
| `subject` | Subject | Yes | Budgeting scope |
| `action` | Action | Yes | Action being budgeted |
| `estimate` | Amount | Yes | Estimated cost |
| `ttl_ms` | integer | No | Reservation TTL in ms (default: tenant `default_reservation_ttl_ms` or 60000, range: 1000–86400000, capped to tenant `max_reservation_ttl_ms`) |
| `grace_period_ms` | integer | No | Grace period after TTL for late commits (default: 5000, range: 0–60000) |
| `overage_policy` | string | No | `REJECT`, `ALLOW_IF_AVAILABLE`, or `ALLOW_WITH_OVERDRAFT` (default: tenant `default_commit_overage_policy` or `ALLOW_IF_AVAILABLE`) |
| `dry_run` | boolean | No | If true, evaluate without reserving (default: false) |
| `metadata` | object | No | Arbitrary key-value metadata |

### Response (200 OK)

```json
{
  "reservation_id": "res-abc-123",
  "decision": "ALLOW",
  "expires_at_ms": 1710000060000,
  "affected_scopes": [
    "tenant:acme",
    "tenant:acme/workspace:production"
  ],
  "scope_path": "tenant:acme/workspace:production",
  "reserved": { "amount": 5000, "unit": "USD_MICROCENTS" },
  "balances": [
    {
      "scope": "tenant:acme",
      "scope_path": "tenant:acme",
      "remaining": { "amount": 95000, "unit": "USD_MICROCENTS" },
      "allocated": { "amount": 100000, "unit": "USD_MICROCENTS" },
      "spent": { "amount": 0, "unit": "USD_MICROCENTS" },
      "reserved": { "amount": 5000, "unit": "USD_MICROCENTS" },
      "debt": { "amount": 0, "unit": "USD_MICROCENTS" },
      "overdraft_limit": { "amount": 0, "unit": "USD_MICROCENTS" },
      "is_over_limit": false
    }
  ],
  "caps": null,
  "reason_code": null,
  "retry_after_ms": null
}
```

When `decision` is `ALLOW_WITH_CAPS`, the `caps` field contains soft constraints.

When `decision` is `DENY` (dry_run only), the reservation is not created. For live reservations, insufficient budget returns a `409` error instead of `decision: DENY`.

When `reason_code` is present (on DENY), it provides a machine-readable reason for the denial. `retry_after_ms` optionally suggests when to retry.

### Dry run response

When `dry_run: true`, the response has the same structure but no reservation is persisted. The `reservation_id` and `expires_at_ms` fields are absent. The `affected_scopes` field is always populated, even when the decision is DENY.

### Example

```bash
curl -X POST http://localhost:7878/v1/reservations \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: your-api-key" \
  -d '{
    "idempotency_key": "req-001",
    "subject": {
      "tenant": "acme",
      "workspace": "production",
      "app": "chatbot"
    },
    "action": {
      "kind": "llm.completion",
      "name": "gpt-4o"
    },
    "estimate": {
      "amount": 5000,
      "unit": "USD_MICROCENTS"
    },
    "ttl_ms": 60000,
    "overage_policy": "REJECT"
  }'
```

### Error responses

| Code | Error | When |
|---|---|---|
| 400 | `INVALID_REQUEST` | Missing or invalid fields |
| 400 | `UNIT_MISMATCH` | `estimate.unit` does not match any budget at the derived scopes (a budget exists in a different unit) |
| 401 | `UNAUTHORIZED` | Missing or invalid API key |
| 403 | `FORBIDDEN` | Tenant mismatch |
| 404 | `NOT_FOUND` | No budget ledger exists at any derived scope in any unit (message: `"Budget not found for provided scope: ..."`) |
| 409 | `BUDGET_EXCEEDED` | Insufficient budget |
| 409 | `BUDGET_FROZEN` | Budget scope is frozen |
| 409 | `BUDGET_CLOSED` | Budget scope is permanently closed |
| 409 | `OVERDRAFT_LIMIT_EXCEEDED` | Scope is over-limit |
| 409 | `DEBT_OUTSTANDING` | Scope has unpaid debt (no overdraft limit configured) |
| 409 | `TENANT_CLOSED` | Owning tenant's status is `CLOSED` (persisting create only, `dry_run` absent or `false`; deployments with a governance plane — spec v0.1.25.13, cycles-server 0.1.25.47+) |
| 409 | `IDEMPOTENCY_MISMATCH` | Same key, different payload |

**Dry run:** when `dry_run=true`, budget-state conditions (`BUDGET_EXCEEDED`, `BUDGET_FROZEN`, `BUDGET_CLOSED`, `OVERDRAFT_LIMIT_EXCEEDED`, `DEBT_OUTSTANDING`, `TENANT_CLOSED` on a closed owning tenant, and the 404 "no budget at any scope" case) surface as `200 OK` with `decision: DENY` and a `reason_code` field — `DecisionReasonCode` is an open string (as of v0.1.25); clients MUST handle unknown values gracefully — not as 4xx/409 errors. Request-validity errors (`INVALID_REQUEST`, `UNIT_MISMATCH`, `UNAUTHORIZED`, `FORBIDDEN`, `IDEMPOTENCY_MISMATCH`) are still returned as 4xx on dry-run. See [Decision reason codes](/protocol/error-codes-and-error-handling-in-cycles#decision-reason-codes).

---

## POST /v1/reservations/{id}/commit

Record actual usage and release the unused remainder.

### Request body

| Field | Type | Required | Description |
|---|---|---|---|
| `idempotency_key` | string | Yes | Unique key for idempotent retries |
| `actual` | Amount | Yes | Actual cost consumed |
| `metrics` | object | No | Standard metrics (see below) |
| `metadata` | object | No | Arbitrary audit metadata |

#### Metrics object

```json
{
  "tokens_input": 150,
  "tokens_output": 80,
  "latency_ms": 320,
  "model_version": "gpt-4o-2024-08-06",
  "custom": { "cache_hit": true }
}
```

### Response (200 OK)

```json
{
  "status": "COMMITTED",
  "charged": { "amount": 3200, "unit": "USD_MICROCENTS" },
  "released": { "amount": 1800, "unit": "USD_MICROCENTS" },
  "balances": [
    {
      "scope": "tenant:acme",
      "scope_path": "tenant:acme",
      "remaining": { "amount": 96800, "unit": "USD_MICROCENTS" },
      "allocated": { "amount": 100000, "unit": "USD_MICROCENTS" },
      "spent": { "amount": 3200, "unit": "USD_MICROCENTS" },
      "reserved": { "amount": 0, "unit": "USD_MICROCENTS" },
      "debt": { "amount": 0, "unit": "USD_MICROCENTS" },
      "overdraft_limit": { "amount": 0, "unit": "USD_MICROCENTS" },
      "is_over_limit": false
    }
  ]
}
```

### Example

```bash
curl -X POST http://localhost:7878/v1/reservations/res-abc-123/commit \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: your-api-key" \
  -d '{
    "idempotency_key": "commit-001",
    "actual": {
      "amount": 3200,
      "unit": "USD_MICROCENTS"
    },
    "metrics": {
      "tokens_input": 150,
      "tokens_output": 80,
      "latency_ms": 320
    }
  }'
```

### Error responses

| Code | Error | When |
|---|---|---|
| 400 | `UNIT_MISMATCH` | Commit unit differs from reservation unit |
| 401 | `UNAUTHORIZED` | Missing or invalid API key |
| 403 | `FORBIDDEN` | Reservation owned by different tenant |
| 404 | `NOT_FOUND` | Reservation does not exist |
| 409 | `BUDGET_EXCEEDED` | Actual exceeds budget (REJECT only) |
| 409 | `BUDGET_FROZEN` | Budget scope is frozen |
| 409 | `BUDGET_CLOSED` | Budget scope is permanently closed |
| 409 | `OVERDRAFT_LIMIT_EXCEEDED` | Debt would exceed limit (ALLOW_WITH_OVERDRAFT) |
| 409 | `RESERVATION_FINALIZED` | Already committed or released |
| 409 | `TENANT_CLOSED` | Owning tenant's status is `CLOSED` (spec v0.1.25.13, cycles-server 0.1.25.47+); takes precedence over reservation-state errors for non-replay requests |
| 409 | `IDEMPOTENCY_MISMATCH` | Same key, different payload |
| 410 | `RESERVATION_EXPIRED` | TTL + grace period elapsed |

---

## POST /v1/reservations/{id}/release

Cancel a reservation and return all reserved budget to the pool.

### Request body

| Field | Type | Required | Description |
|---|---|---|---|
| `idempotency_key` | string | Yes | Unique key for idempotent retries |
| `reason` | string | No | Human-readable reason for release |

### Response (200 OK)

```json
{
  "status": "RELEASED",
  "released": { "amount": 5000, "unit": "USD_MICROCENTS" },
  "balances": [
    {
      "scope": "tenant:acme",
      "scope_path": "tenant:acme",
      "remaining": { "amount": 100000, "unit": "USD_MICROCENTS" },
      "allocated": { "amount": 100000, "unit": "USD_MICROCENTS" },
      "spent": { "amount": 0, "unit": "USD_MICROCENTS" },
      "reserved": { "amount": 0, "unit": "USD_MICROCENTS" },
      "debt": { "amount": 0, "unit": "USD_MICROCENTS" },
      "overdraft_limit": { "amount": 0, "unit": "USD_MICROCENTS" },
      "is_over_limit": false
    }
  ]
}
```

### Example

```bash
curl -X POST http://localhost:7878/v1/reservations/res-abc-123/release \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: your-api-key" \
  -d '{
    "idempotency_key": "release-001",
    "reason": "Task cancelled by user"
  }'
```

### Error responses

| Code | Error | When |
|---|---|---|
| 400 | `INVALID_REQUEST` | Malformed request (e.g., missing `idempotency_key`) |
| 401 | `UNAUTHORIZED` | Missing or invalid API key |
| 403 | `FORBIDDEN` | Reservation owned by different tenant |
| 404 | `NOT_FOUND` | Reservation does not exist |
| 409 | `RESERVATION_FINALIZED` | Already committed or released |
| 409 | `TENANT_CLOSED` | Owning tenant's status is `CLOSED` (spec v0.1.25.13, cycles-server 0.1.25.47+); takes precedence over reservation-state errors for non-replay requests |
| 409 | `IDEMPOTENCY_MISMATCH` | Same key, different payload |
| 410 | `RESERVATION_EXPIRED` | TTL + grace period elapsed |

---

## POST /v1/reservations/{id}/extend

Extend the TTL of an active reservation. Used as a heartbeat for long-running operations.

### Request body

| Field | Type | Required | Description |
|---|---|---|---|
| `idempotency_key` | string | Yes | Unique key for idempotent retries |
| `extend_by_ms` | integer | Yes | Milliseconds to extend (range: 1–86400000) |
| `metadata` | object | No | Optional debugging/audit metadata |

### Response (200 OK)

| Field | Type | Required | Description |
|---|---|---|---|
| `status` | string | Yes | Always `"ACTIVE"` after a successful extension |
| `expires_at_ms` | integer (int64) | Yes | New server-authoritative expiry timestamp (ms) |
| `balances` | array of Balance | No | Optional updated balances snapshot after extension |

```json
{
  "status": "ACTIVE",
  "expires_at_ms": 1710000120000,
  "balances": []
}
```

### Example

```bash
curl -X POST http://localhost:7878/v1/reservations/res-abc-123/extend \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: your-api-key" \
  -d '{
    "idempotency_key": "extend-001",
    "extend_by_ms": 60000
  }'
```

### Error responses

| Code | Error | When |
|---|---|---|
| 400 | `INVALID_REQUEST` | Missing or invalid fields |
| 401 | `UNAUTHORIZED` | Missing or invalid API key |
| 403 | `FORBIDDEN` | Reservation owned by different tenant |
| 404 | `NOT_FOUND` | Reservation does not exist |
| 409 | `RESERVATION_FINALIZED` | Already committed or released |
| 409 | `TENANT_CLOSED` | Owning tenant's status is `CLOSED` (spec v0.1.25.13, cycles-server 0.1.25.47+); takes precedence over reservation-state errors for non-replay requests |
| 409 | `IDEMPOTENCY_MISMATCH` | Same key, different payload |
| 409 | `MAX_EXTENSIONS_EXCEEDED` | Tenant `max_reservation_extensions` limit reached |
| 410 | `RESERVATION_EXPIRED` | Past TTL (no grace period for extend) |

---

## GET /v1/reservations

List reservations with optional filters and pagination.

### Query parameters

| Parameter | Type | Description |
|---|---|---|
| `tenant` | string | Filter by tenant |
| `workspace` | string | Filter by workspace |
| `app` | string | Filter by app |
| `workflow` | string | Filter by workflow |
| `agent` | string | Filter by agent |
| `toolset` | string | Filter by toolset |
| `status` | string | Filter by status: `ACTIVE`, `COMMITTED`, `RELEASED`, `EXPIRED` |
| `idempotency_key` | string | Filter by idempotency key |
| `from` / `to` | string | ISO 8601 inclusive bounds on `created_at_ms`; either side may be supplied alone |
| `expires_from` / `expires_to` | string | ISO 8601 inclusive bounds on `expires_at_ms`; useful for finding stale or soon-expiring reservations |
| `finalized_from` / `finalized_to` | string | ISO 8601 inclusive bounds on `finalized_at_ms`; only COMMITTED and RELEASED rows match |
| `sort_by` | string | Column to sort by (v0.1.25.12+). See [Sorting](#sorting) below. |
| `sort_dir` | string | `asc` or `desc`. Default `desc`. |
| `include` | string | Comma-separated projection tokens: `metadata`, `committed_metadata`, `evidence` |
| `limit` | integer | Max results (1–200, default: 50) |
| `cursor` | string | Opaque cursor from previous response |

Under `X-Cycles-API-Key`, `tenant` is validation-only and must match the authenticated tenant. Under `X-Admin-API-Key`, `tenant` is required as a filter because admin auth has no effective tenant.

### Sorting

`sort_by` (v0.1.25.12+) accepts one of seven column names:

| Value | Sorts by |
|---|---|
| `reservation_id` | Reservation identifier (lexicographic) |
| `tenant` | Tenant slug |
| `scope_path` | Full scope path |
| `status` | Reservation status enum |
| `reserved` | Held amount (numeric, unit-native) |
| `created_at_ms` | Creation timestamp |
| `expires_at_ms` | Expiry timestamp |

Unknown values return `400 INVALID_REQUEST`. `sort_dir` defaults to `desc`; pass `asc` to reverse.

**Cursor-tuple binding.** The opaque cursor binds to the `(sort_by, sort_dir, filters)` tuple it was issued under. Reusing a cursor with a different sort key, direction, or filter set returns `400 INVALID_REQUEST` — a new first-page request must issue a new cursor. Callers that switch sort mid-walk should discard the cursor and restart.

**Sorted hydration warning.** Current reference servers hydrate all matching rows for sorted reservation listings, then sort and slice. When a sorted query hydrates 2,000 or more rows, the server logs a WARN so operators can add narrower filters or plan sorted indices. Rows beyond 2,000 are no longer truncated in v0.1.25.39+.

### Response (200 OK)

```json
{
  "reservations": [
    {
      "reservation_id": "res-abc-123",
      "status": "ACTIVE",
      "subject": { "tenant": "acme", "workspace": "production" },
      "action": { "kind": "llm.completion", "name": "gpt-4o" },
      "reserved": { "amount": 5000, "unit": "USD_MICROCENTS" },
      "expires_at_ms": 1710000060000,
      "created_at_ms": 1710000000000,
      "scope_path": "tenant:acme/workspace:production",
      "affected_scopes": ["tenant:acme", "tenant:acme/workspace:production"]
    }
  ],
  "has_more": false,
  "next_cursor": null
}
```

### Example

```bash
curl -s "http://localhost:7878/v1/reservations?tenant=acme&status=ACTIVE&include=evidence&limit=10" \
  -H "X-Cycles-API-Key: your-api-key"
```

### Error responses

| Code | Error | When |
|---|---|---|
| 400 | `INVALID_REQUEST` | Invalid filter parameters |
| 401 | `UNAUTHORIZED` | Missing or invalid API key |
| 403 | `FORBIDDEN` | Tenant mismatch under tenant auth |

---

## GET /v1/reservations/{id}

Get details of a specific reservation.

### Response (200 OK)

```json
{
  "reservation_id": "res-abc-123",
  "status": "COMMITTED",
  "idempotency_key": "req-001",
  "subject": { "tenant": "acme", "workspace": "production" },
  "action": { "kind": "llm.completion", "name": "gpt-4o" },
  "reserved": { "amount": 5000, "unit": "USD_MICROCENTS" },
  "committed": { "amount": 3200, "unit": "USD_MICROCENTS" },
  "created_at_ms": 1710000000000,
  "expires_at_ms": 1710000060000,
  "finalized_at_ms": 1710000045000,
  "scope_path": "tenant:acme/workspace:production",
  "affected_scopes": ["tenant:acme", "tenant:acme/workspace:production"],
  "metadata": {},
  "committed_metadata": {},
  "evidence": {
    "reserve": {
      "evidence_id": "8403bed43e13ef7d56a8ab402a9d29ee7dd2f405e24c0cacb51068341a5e7030",
      "cycles_evidence_url": "https://cycles.example.com/v1/evidence/8403bed43e13ef7d56a8ab402a9d29ee7dd2f405e24c0cacb51068341a5e7030"
    }
  }
}
```

### Example

```bash
curl -s http://localhost:7878/v1/reservations/res-abc-123 \
  -H "X-Cycles-API-Key: your-api-key"
```

### Error responses

| Code | Error | When |
|---|---|---|
| 401 | `UNAUTHORIZED` | Missing or invalid API key |
| 403 | `FORBIDDEN` | Reservation owned by different tenant |
| 404 | `NOT_FOUND` | Reservation does not exist |
| 410 | `RESERVATION_EXPIRED` | Reservation has expired |

---

## POST /v1/decide

Evaluate a budget decision without creating a reservation. Useful for preflight checks, UI affordances, and routing decisions.

### Request body

| Field | Type | Required | Description |
|---|---|---|---|
| `idempotency_key` | string | Yes | Unique key for idempotent retries |
| `subject` | Subject | Yes | Budgeting scope |
| `action` | Action | Yes | Action being evaluated |
| `estimate` | Amount | Yes | Estimated cost to evaluate |
| `metadata` | object | No | Arbitrary metadata |

### Response (200 OK)

```json
{
  "decision": "ALLOW",
  "affected_scopes": [
    "tenant:acme",
    "tenant:acme/workspace:production"
  ]
}
```

The `reason_code` and `retry_after_ms` fields are present when the decision is `DENY`. `reason_code` is `DecisionReasonCode` — an open string (as of v0.1.25) with seven documented known values: `BUDGET_EXCEEDED`, `BUDGET_FROZEN`, `BUDGET_CLOSED`, `BUDGET_NOT_FOUND`, `OVERDRAFT_LIMIT_EXCEEDED`, `DEBT_OUTSTANDING`, `TENANT_CLOSED` (added in spec v0.1.25.13 for closed owning tenants). Clients MUST handle unknown values gracefully. See [Decision reason codes](/protocol/error-codes-and-error-handling-in-cycles#decision-reason-codes).

### Example

```bash
curl -X POST http://localhost:7878/v1/decide \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: your-api-key" \
  -d '{
    "idempotency_key": "decide-001",
    "subject": { "tenant": "acme", "workspace": "production" },
    "action": { "kind": "llm.completion", "name": "gpt-4o" },
    "estimate": { "amount": 5000, "unit": "USD_MICROCENTS" }
  }'
```

### Error responses

| Code | Error | When |
|---|---|---|
| 400 | `INVALID_REQUEST` | Missing or invalid fields |
| 400 | `UNIT_MISMATCH` | `estimate.unit` does not match any budget at the derived scopes (a budget exists in a different unit) |
| 401 | `UNAUTHORIZED` | Missing or invalid API key |
| 403 | `FORBIDDEN` | Tenant mismatch |
| 409 | `IDEMPOTENCY_MISMATCH` | Same key, different payload |

Note: decide returns `200` with `decision: DENY` for all budget-state conditions — insufficient remaining, debt, overdraft, frozen, closed, the "no budget exists at any scope" case, and a closed owning tenant — not a `409` or `404`. The specific reason is surfaced in the `reason_code` field. `DecisionReasonCode` is an open string (as of v0.1.25) with seven documented known values: `BUDGET_EXCEEDED`, `BUDGET_FROZEN`, `BUDGET_CLOSED`, `BUDGET_NOT_FOUND`, `OVERDRAFT_LIMIT_EXCEEDED`, `DEBT_OUTSTANDING`, `TENANT_CLOSED` (spec v0.1.25.13; fresh evaluations on a closed owning tenant — never `409 TENANT_CLOSED` on this endpoint, though a present-but-malformed tenant record fails closed with `500 INTERNAL_ERROR`). Clients MUST handle unknown values gracefully. See [Decision reason codes](/protocol/error-codes-and-error-handling-in-cycles#decision-reason-codes) for full semantics. Request-validity errors like `UNIT_MISMATCH` are still returned as `400`.

---

## GET /v1/balances

Query current budget state for one or more scopes.

### Query parameters

| Parameter | Type | Description |
|---|---|---|
| `tenant` | string | Filter by tenant |
| `workspace` | string | Filter by workspace |
| `app` | string | Filter by app |
| `workflow` | string | Filter by workflow |
| `agent` | string | Filter by agent |
| `toolset` | string | Filter by toolset |
| `include_children` | boolean | Include child scopes (default: false) |
| `limit` | integer | Max results (1–200, default: 50) |
| `cursor` | string | Opaque cursor from previous response |

At least one of `tenant`, `workspace`, `app`, `workflow`, `agent`, or `toolset` must be provided. The `tenant` parameter is validation-only: if provided, it must match the effective tenant derived from the API key.

### Response (200 OK)

```json
{
  "balances": [
    {
      "scope": "tenant:acme",
      "scope_path": "tenant:acme",
      "remaining": { "amount": 96800, "unit": "USD_MICROCENTS" },
      "allocated": { "amount": 100000, "unit": "USD_MICROCENTS" },
      "spent": { "amount": 3200, "unit": "USD_MICROCENTS" },
      "reserved": { "amount": 0, "unit": "USD_MICROCENTS" },
      "debt": { "amount": 0, "unit": "USD_MICROCENTS" },
      "overdraft_limit": { "amount": 0, "unit": "USD_MICROCENTS" },
      "is_over_limit": false
    },
    {
      "scope": "workspace:production",
      "scope_path": "tenant:acme/workspace:production",
      "remaining": { "amount": 46800, "unit": "USD_MICROCENTS" },
      "allocated": { "amount": 50000, "unit": "USD_MICROCENTS" },
      "spent": { "amount": 3200, "unit": "USD_MICROCENTS" },
      "reserved": { "amount": 0, "unit": "USD_MICROCENTS" },
      "debt": { "amount": 0, "unit": "USD_MICROCENTS" },
      "overdraft_limit": { "amount": 0, "unit": "USD_MICROCENTS" },
      "is_over_limit": false
    }
  ],
  "has_more": false,
  "next_cursor": null
}
```

### Example

```bash
curl -s "http://localhost:7878/v1/balances?tenant=acme&workspace=production" \
  -H "X-Cycles-API-Key: your-api-key"
```

### Error responses

| Code | Error | When |
|---|---|---|
| 400 | `INVALID_REQUEST` | No subject filter provided |
| 401 | `UNAUTHORIZED` | Missing or invalid API key |
| 403 | `FORBIDDEN` | Tenant mismatch |

---

## POST /v1/events

Record a direct debit event without a prior reservation. Used for post-hoc accounting when the reserve → commit lifecycle does not apply.

### Request body

| Field | Type | Required | Description |
|---|---|---|---|
| `idempotency_key` | string | Yes | Unique key for idempotent retries |
| `subject` | Subject | Yes | Budgeting scope |
| `action` | Action | Yes | Action being recorded |
| `actual` | Amount | Yes | Actual cost to record |
| `overage_policy` | string | No | `REJECT`, `ALLOW_IF_AVAILABLE`, or `ALLOW_WITH_OVERDRAFT` (default: tenant `default_commit_overage_policy` or `ALLOW_IF_AVAILABLE`) |
| `metrics` | object | No | Standard metrics |
| `client_time_ms` | integer | No | Client-side timestamp |
| `metadata` | object | No | Arbitrary metadata |

### Response (201 Created)

```json
{
  "status": "APPLIED",
  "event_id": "evt-abc-123",
  "charged": { "amount": 4400, "unit": "USD_MICROCENTS" },
  "balances": [
    {
      "scope": "tenant:acme",
      "scope_path": "tenant:acme",
      "remaining": { "amount": 95600, "unit": "USD_MICROCENTS" },
      "allocated": { "amount": 100000, "unit": "USD_MICROCENTS" },
      "spent": { "amount": 4400, "unit": "USD_MICROCENTS" },
      "reserved": { "amount": 0, "unit": "USD_MICROCENTS" },
      "debt": { "amount": 0, "unit": "USD_MICROCENTS" },
      "overdraft_limit": { "amount": 0, "unit": "USD_MICROCENTS" },
      "is_over_limit": false
    }
  ]
}
```

`charged` is optional. It is present when `overage_policy` is `ALLOW_IF_AVAILABLE` and the actual amount was capped to the remaining budget, so the client can see the effective charge applied.

### Example

```bash
curl -X POST http://localhost:7878/v1/events \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: your-api-key" \
  -d '{
    "idempotency_key": "evt-001",
    "subject": {
      "tenant": "acme",
      "workspace": "production"
    },
    "action": {
      "kind": "search.api",
      "name": "google-search"
    },
    "actual": {
      "amount": 1200,
      "unit": "USD_MICROCENTS"
    }
  }'
```

### Error responses

| Code | Error | When |
|---|---|---|
| 400 | `INVALID_REQUEST` | Missing or invalid fields |
| 400 | `UNIT_MISMATCH` | `actual.unit` does not match any budget at the target scope (a budget exists in a different unit) |
| 401 | `UNAUTHORIZED` | Missing or invalid API key |
| 403 | `FORBIDDEN` | Tenant mismatch |
| 404 | `NOT_FOUND` | No budget ledger exists at any derived scope in any unit (message: `"Budget not found for provided scope: ..."`) |
| 409 | `BUDGET_EXCEEDED` | Insufficient budget (REJECT only) |
| 409 | `BUDGET_FROZEN` | Budget scope is frozen |
| 409 | `BUDGET_CLOSED` | Budget scope is permanently closed |
| 409 | `OVERDRAFT_LIMIT_EXCEEDED` | Debt would exceed limit |
| 409 | `IDEMPOTENCY_MISMATCH` | Same key, different payload |

---

## GET /v1/evidence/{evidence_id}

Fetch a signed CyclesEvidence envelope by content id. This endpoint is public: the `evidence_id` is a 64-character lowercase SHA-256 content hash carried by a prior `cycles_evidence` response reference, and the returned envelope is content-addressed and signature-verifiable.

### Path parameters

| Parameter | Type | Required | Description |
|---|---|---|---|
| `evidence_id` | string | Yes | 64 lowercase hex characters; SHA-256 content hash of the evidence envelope |

### Response (200 OK)

Returns the signed envelope as JSON. The exact envelope shape is covered in [CyclesEvidence Envelopes](/protocol/cycles-evidence-envelopes-in-cycles).

```json
{
  "schema_version": "cycles-evidence/v0.1",
  "artifact_type": "reserve",
  "server_id": "https://cycles.example.com/v1",
  "signer_did": "b10554...",
  "issued_at_ms": 1781436904050,
  "trace_id": "b2a0ab88...",
  "payload": { "reserve": { "request": {}, "response": {} } },
  "evidence_id": "8403bed43e13ef7d56a8ab402a9d29ee7dd2f405e24c0cacb51068341a5e7030",
  "signature": "4bc8cb9a..."
}
```

### Example

```bash
curl -s http://localhost:7878/v1/evidence/8403bed43e13ef7d56a8ab402a9d29ee7dd2f405e24c0cacb51068341a5e7030
```

### Error responses

| Code | Error | When |
|---|---|---|
| 400 | `INVALID_REQUEST` | `evidence_id` is not a valid 64-character lowercase hex string |
| 404 | `NOT_FOUND` | Envelope is not available or evidence signing/storage is not configured |
| 429 | `LIMIT_EXCEEDED` | Public endpoint throttled (reference server default: 300 requests/minute per client IP); retry after `Retry-After` |

---

## GET /v1/.well-known/cycles-jwks.json

Fetch the issuing server's public CyclesEvidence JWK Set. This endpoint is public and contains verification keys only; the private signing key is never served.

The path is API-base-relative. If `server_id` is `https://cycles.example.com/v1`, the JWKS URL is `https://cycles.example.com/v1/.well-known/cycles-jwks.json`.

### Response (200 OK)

```json
{
  "keys": [
    {
      "kty": "OKP",
      "crv": "Ed25519",
      "alg": "EdDSA",
      "x": "base64url-public-key",
      "kid": "2026-h2",
      "cycles_nbf_ms": 1781000000000,
      "status": "active"
    }
  ]
}
```

### Example

```bash
curl -s http://localhost:7878/v1/.well-known/cycles-jwks.json
```

### Error responses

| Code | Error | When |
|---|---|---|
| 404 | `NOT_FOUND` | The server does not publish a JWK Set, usually because signer-key resolution is not configured |
| 429 | `LIMIT_EXCEEDED` | Public endpoint throttled (reference server default: 300 requests/minute per client IP); retry after `Retry-After` |

---

## Idempotency

All write operations require idempotency via the `idempotency_key` field in the request body. The `X-Idempotency-Key` header is also accepted; if both are provided, they must match.

- If you retry a request with the same key and the same payload, you get the original successful response. The operation is not applied again.
- If you reuse a key with a different payload, you get `409 IDEMPOTENCY_MISMATCH`.
- If the original request failed, retrying with the same key sends a fresh request.

Idempotency is scoped per (effective tenant, endpoint, idempotency_key).

## Next steps

- [Error Codes and Error Handling](/protocol/error-codes-and-error-handling-in-cycles) — detailed error code reference
- [Self-Hosting the Cycles Server](/quickstart/self-hosting-the-cycles-server) — deploy your own instance
- [Getting Started with the Spring Boot Starter](/quickstart/getting-started-with-the-cycles-spring-boot-starter) — client integration



# Authentication, Tenancy, and API Keys in Cycles

Every protected budget operation is authenticated and tenant-scoped. Liveness/readiness and other explicitly public operational or evidence endpoints are exceptions.

These two properties — authentication and tenancy — are foundational. They determine who is making the request, which budgets are visible, and which reservations can be accessed.

## How authentication works

Cycles authenticates requests using the `X-Cycles-API-Key` header.

Tenant-authenticated runtime requests must include this header. If it is missing or the key is invalid, the server returns `401 UNAUTHORIZED`. Admin and public endpoints follow their own authentication rules.

```
X-Cycles-API-Key: your-api-key
```

There is no session, token exchange, or OAuth flow in the reference API. Protected tenant requests use one API-key header; admin-only routes use `X-Admin-API-Key`.

### Public endpoints (no API key)

The runtime YAML declares two public evidence endpoints with `security: []`:

- `GET /v1/evidence/{evidence_id}` — signed-envelope retrieval. The `evidence_id` is an unguessable content-hash capability, and the envelope is content-addressed and signed, so public read cannot forge or alter it.
- `GET /v1/.well-known/cycles-jwks.json` — the signer's public JWK Set (public keys only, the standard posture for a verification key set), used to verify evidence signatures.

The reference server also leaves `/actuator/health/liveness` and `/actuator/health/readiness` public for probes. Aggregate health, metrics, API documentation, and the remaining operational/API routes require the configured tenant or admin credential.

## The effective tenant

From the API key (or other auth context), the server determines an **effective tenant**.

The effective tenant is the identity that governs all budget operations for that request:

- which budgets can be queried
- which reservations can be created
- which reservations can be committed, released, or extended
- which balances are visible

The effective tenant is not sent by the client. It is derived by the server from the authentication context.

## Tenant validation on every request

Every request that includes a `Subject` must have a `tenant` field that matches the effective tenant.

If the client sends `subject.tenant = "acme"` but the API key maps to tenant `"beta"`, the server returns `403 FORBIDDEN`.

This is a normative rule: the server **must** reject any request where `subject.tenant` does not match the effective tenant.

This prevents one tenant from creating reservations, recording events, or querying budgets against another tenant's scopes.

## Reservation ownership

Every reservation is bound to the effective tenant at creation time.

Any subsequent operation on that reservation — commit, release, extend, or get — must come from the same effective tenant.

If a different tenant attempts to access the reservation, the server returns `403 FORBIDDEN`, even if the reservation ID is known.

This means:

- tenant A cannot commit tenant B's reservation
- tenant A cannot release tenant B's reservation
- tenant A cannot extend tenant B's reservation
- tenant A cannot view tenant B's reservation details

Reservation ownership is enforced at the protocol level, not by convention.

## Balance visibility

Balance queries are tenant-scoped.

The server only returns balances within the effective tenant's scope. If the `tenant` query parameter is provided, it is validation-only — it must match the effective tenant, or the server returns `403 FORBIDDEN`.

If the `tenant` parameter is omitted, the effective tenant is used automatically.

A tenant cannot query another tenant's balances under any circumstances.

## Tenant validation on listing endpoints

The reservation listing endpoint (`GET /v1/reservations`) follows the same tenancy rules:

- results are scoped to the effective tenant
- if the `tenant` query parameter is provided, it must match the effective tenant
- if it does not match, the server returns `403 FORBIDDEN`

This ensures that listing and recovery operations are always tenant-isolated.

## Admin access on behalf of a tenant

Three runtime reservation endpoints also accept the admin key header `X-Admin-API-Key` (the `AdminKeyAuth` scheme — the same header the governance-admin spec uses), so admin operators can authenticate against the runtime plane with one key:

- `GET /v1/reservations` (list) — the admin caller has no effective tenant, so the `tenant` query parameter is **required** and is used as a filter, not validation. Omitting it returns `400 INVALID_REQUEST`.
- `GET /v1/reservations/{reservation_id}` (get) — admin operators can read any reservation regardless of owning tenant; the reservation ID already pins the owner, so no extra parameter is needed.
- `POST /v1/reservations/{reservation_id}/release` — admin operators can release any reservation regardless of owning tenant (the ops use case is force-expiring a hung reservation during incident response). The audit-log entry for an admin-driven release must record `actor_type=admin_on_behalf_of`, so security review can distinguish admin-driven releases from tenant self-service releases.

Create, commit, and extend do not accept the admin key — they remain tenant-key-only operations.

## The decide endpoint and tenancy

The decide endpoint (`POST /v1/decide`) follows the same rule: `subject.tenant` must match the effective tenant.

Even though decide is a read-only preflight check that does not modify budget state, it still enforces tenant isolation.

## How the X-Cycles-Tenant response header works

The server may include an `X-Cycles-Tenant` response header on any response.

This header contains the effective tenant identifier derived from the authentication context.

It is useful for:

- debugging tenant mismatch errors
- confirming which tenant the server resolved from the API key
- logging and correlation in multi-tenant environments

This header is optional in v0 implementations.

## API key permissions (governance layer)

When using the Cycles governance server (admin API), API keys can carry granular permissions that restrict which operations they can perform. The governance spec's `Permission` enum defines 27 values in three groups.

### Tenant permissions (13)

Tenant-scoped keys should carry only these permissions:

| Permission | Operations |
|---|---|
| `reservations:create` | Create reservations |
| `reservations:commit` | Commit reservations |
| `reservations:release` | Release reservations |
| `reservations:extend` | Extend reservations |
| `reservations:list` | List and get reservations |
| `balances:read` | Query balances |
| `budgets:read` | List and read budgets |
| `budgets:write` | Create, update, and fund budgets |
| `policies:read` | List and read policies |
| `policies:write` | Create and update policies |
| `webhooks:read` | Read webhook subscriptions and deliveries (self-service) |
| `webhooks:write` | Create, update, and delete webhook subscriptions (self-service) |
| `events:read` | Read the tenant's event stream |

### Legacy wildcard admin permissions (2)

`admin:read` and `admin:write` are reserved for backward compatibility with legacy keys and SHOULD NOT be assigned to new tenant keys — use the granular tenant permissions above instead. Their wildcard semantics are normative:

- `admin:write` satisfies any `*:write` requirement (`budgets:write`, `policies:write`, `webhooks:write`, etc.)
- `admin:read` satisfies any `*:read` requirement (`budgets:read`, `policies:read`, `events:read`, `balances:read`, etc.)
- `admin:read` does **not** satisfy `*:write`

### Granular admin permissions (12)

Admin-plane operations use granular admin permissions rather than the legacy wildcards: `admin:tenants:read`, `admin:tenants:write`, `admin:budgets:read`, `admin:budgets:write`, `admin:policies:read`, `admin:policies:write`, `admin:apikeys:read`, `admin:apikeys:write`, `admin:webhooks:read`, `admin:webhooks:write`, `admin:events:read`, and `admin:audit:read`.

### Default permission set

When `permissions` is omitted from the API key creation request, the server assigns the default set for tenant keys — 10 permissions: `[reservations:create, reservations:commit, reservations:release, reservations:extend, reservations:list, balances:read, budgets:read, budgets:write, policies:read, policies:write]`.

Only the webhook and event self-service permissions (`webhooks:read`, `webhooks:write`, `events:read`) are excluded from the default set and must be explicitly granted.

If a request requires a permission the API key does not have, the governance server returns `403` with error code `INSUFFICIENT_PERMISSIONS` (defined in the governance spec, not the protocol spec).

### Key prefixes

Tenant keys are issued in the format `cyc_live_{random}` or `cyc_test_{random}`. Only the visible `key_prefix` (e.g., `cyc_live_abc123`) is returned for identification after creation — the full key secret is returned exactly once, at creation time, and is stored server-side only as a hash.

## Scope filter (governance layer)

API keys can optionally be restricted to specific scopes using a **scope filter**.

A scope filter is an array of scope segment patterns:

```json
"scope_filter": ["workspace:eng", "agent:*"]
```

When a scope filter is set on an API key:

- operations are only permitted on scopes containing a matching segment
- `"workspace:eng"` matches scopes like `tenant:acme/workspace:eng/agent:bot1` (exact segment match)
- `"agent:*"` matches any scope containing an agent segment (wildcard match)
- if no filter entry matches the requested scope, the server returns `403 FORBIDDEN`

When scope filter is empty or not set, the key has unrestricted access within its tenant.

## Practical implications

### One API key per tenant (typical)

Most deployments map each API key to exactly one tenant. This makes tenant validation automatic — the client sets `subject.tenant` to match its key, and all operations are scoped correctly.

### Multi-tenant API keys (advanced)

Some deployments may use API keys that can operate on behalf of multiple tenants. In this case, the effective tenant derivation logic is implementation-specific and may involve additional headers or request context.

The protocol does not define how effective tenant is derived — only that it must be derived and enforced consistently.

### Tenant mismatch debugging

When a `403 FORBIDDEN` error occurs, the most common cause is a tenant mismatch:

- the `subject.tenant` in the request does not match the effective tenant from the API key
- a commit or release targets a reservation owned by a different tenant
- a balance query specifies a tenant that does not match the API key

Check the `X-Cycles-Tenant` response header (if present) to see which tenant the server resolved.

## Security properties

The authentication and tenancy model provides several guarantees:

- **Isolation**: tenants cannot see or modify each other's budgets, reservations, or balances
- **Ownership**: reservations are permanently bound to the creating tenant
- **Validation**: every protected tenant operation is checked against the effective tenant before processing
- **Consistency**: the same tenancy rules apply to all endpoints (reserve, commit, release, extend, decide, events, balances, listing)

These properties hold regardless of whether the client is trusted. The server enforces them on protected tenant operations.

## Summary

Tenant-plane authentication uses the `X-Cycles-API-Key` header. Admin-only routes use `X-Admin-API-Key`, and explicitly public routes require neither.

The server derives an effective tenant from the key and enforces tenant isolation across all operations:

- **Subject.tenant** must match the effective tenant on operations that carry a `Subject`; tenant query parameters are validation-only where supported
- **Reservation ownership** is enforced on commit, release, extend, and get
- **Balance visibility** is scoped to the effective tenant
- **403 FORBIDDEN** is returned for any tenant mismatch

This ensures that budget governance is always tenant-isolated, even in shared deployments.

## Next steps

- [Tenant Creation and Management](/how-to/tenant-creation-and-management-in-cycles) — create, configure, and manage tenants via the Admin API
- [API Key Management](/how-to/api-key-management-in-cycles) — create and rotate tenant-scoped API keys
- [Budget Allocation and Management](/how-to/budget-allocation-and-management-in-cycles) — set up budgets at tenant and sub-scopes
- [Scope Derivation](/protocol/how-scope-derivation-works-in-cycles) — how tenant scopes fit into the budget hierarchy
- [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack) — set up the Cycles infrastructure from scratch
- Integrate with [Python](/quickstart/getting-started-with-the-python-client), [TypeScript](/quickstart/getting-started-with-the-typescript-client), or [Spring AI](/how-to/integrating-cycles-with-spring-ai)



# Caps and the Three-Way Decision Model in Cycles

Most budget systems make a binary decision: allow or deny.

Cycles adds a third option.

When a reservation or decide request is evaluated, the server returns one of three decisions:

- **ALLOW** — sufficient budget exists, proceed normally
- **ALLOW_WITH_CAPS** — sufficient budget exists, but soft constraints apply
- **DENY** — insufficient budget or policy block

The middle option — ALLOW_WITH_CAPS — is what makes Cycles more useful than a simple gate.

## Where DENY appears

One wire-level detail matters here: `decision: DENY` only ever appears on responses that do not hold budget — `POST /v1/decide` responses and dry-run (`dry_run: true`) reservation responses.

A live (non-dry-run) reservation never returns `decision: DENY`. When budget is unavailable, the server rejects the request with an HTTP `409` error — `BUDGET_EXCEEDED`, or another 409 code such as `OVERDRAFT_LIMIT_EXCEEDED`, `DEBT_OUTSTANDING`, or `TENANT_CLOSED` — instead of a 200 response carrying a DENY decision.

Where callers find the denial reason follows the same split. On a DENY decision (decide or dry run), the response's `reason_code` field carries the machine-readable reason — `BUDGET_EXCEEDED`, `BUDGET_FROZEN`, `BUDGET_CLOSED`, `BUDGET_NOT_FOUND`, `OVERDRAFT_LIMIT_EXCEEDED`, `DEBT_OUTSTANDING`, or `TENANT_CLOSED`. On a live denial, the equivalent information is in the 409 error response's `error` field. See [Decision reason codes](/protocol/error-codes-and-error-handling-in-cycles#decision-reason-codes).

## Why binary decisions are not enough

In practice, many autonomous actions can still produce value in a constrained mode.

A model call does not have to use the maximum token budget.
An agent does not have to invoke every available tool.
A workflow does not have to run at full concurrency.

If the only options are "full speed" or "stop," teams end up choosing one extreme:

- budgets are set too loose (to avoid breaking production)
- or budgets are set too tight (and useful work gets denied unnecessarily)

Caps provide a middle ground.

## What caps are

Caps are operator-configured constraints that the server returns from the deepest matching budget when the decision is ALLOW_WITH_CAPS.

The protocol defines five cap fields:

### max_tokens

The maximum number of tokens the action should consume.

This lets the server say: "you may proceed, but limit output length."

### max_steps_remaining

The maximum number of remaining steps for an agent or loop.

This lets the server say: "you may proceed, but wrap up soon."

### tool_allowlist

A list of tools the action is allowed to use.

If present, only these tools may be invoked. All others are implicitly denied.

### tool_denylist

A list of tools the action is not allowed to use.

If present, all tools except these may be invoked.

The allowlist takes precedence: if `tool_allowlist` is non-empty, the denylist is ignored.

### cooldown_ms

A delay in milliseconds before the next action should begin.

This lets the server throttle execution without denying it outright.

## How caps flow through the system

Caps are returned by the server in two places:

1. **Reservation responses** — when the decision on `POST /v1/reservations` is ALLOW_WITH_CAPS
2. **Decide responses** — when the decision on `POST /v1/decide` is ALLOW_WITH_CAPS

Caps are only present when the decision is ALLOW_WITH_CAPS. They are absent for ALLOW and DENY.

The client is responsible for respecting caps. The server does not enforce them at commit time. They are guidance, not hard blocks.

## When caps appear

In the current server, caps come from the deepest matching budget that has an operator-configured `caps` object. Their presence is configuration-driven, not an automatic response to utilization or remaining balance. A deployment that wants progressive narrowing must change which configured policy or scope applies; the base server does not tighten caps as spend rises.

## Using caps in practice

### Model calls

When `max_tokens` is returned, the client should pass it to the model provider as a generation limit.

This reduces the cost of the call without denying it entirely.

### Agent loops

When `max_steps_remaining` is returned, the agent should plan to finish within that many steps.

This creates a bounded wind-down instead of an abrupt stop.

### Tool selection

When `tool_allowlist` or `tool_denylist` is returned, the agent should filter its available tools accordingly.

This narrows the action surface when the configured policy calls for it without eliminating all capability.

### Pacing

When `cooldown_ms` is returned, the client should wait that long before making the next action.

This reduces execution velocity without stopping the workflow.

## Caps in client code

Inside a `@cycles`-decorated function (Python) or `@Cycles`-annotated method (Java), access caps through the context object:

::: code-group
```python [Python]
from runcycles import cycles, get_cycles_context

@cycles(estimate=1000)
def process(prompt: str) -> str:
    ctx = get_cycles_context()
    if ctx.has_caps():
        if ctx.caps.max_tokens:
            # limit generation length
            pass
        if not ctx.caps.is_tool_allowed("web.search"):
            # skip web search tool
            pass
    return call_llm(prompt)
```
```java [Java (Spring Boot)]
CyclesReservationContext ctx = CyclesContextHolder.get();
if (ctx.hasCaps()) {
    Caps caps = ctx.getCaps();
    if (caps.getMaxTokens() != null) {
        // limit generation length
    }
    if (!caps.isToolAllowed("web.search")) {
        // skip web search tool
    }
}
```
:::

The `@Cycles` annotation logs a warning when ALLOW_WITH_CAPS is returned, so teams can see when caps are being applied even without explicit handling.

## Caps are advisory

Caps are not enforced by the server at commit time.

If the client ignores `max_tokens` and generates more output, the commit will still succeed (subject to the overage policy).

But respecting caps is important for two reasons:

1. It keeps budget consumption aligned with server expectations
2. Lower-cost execution can preserve remaining budget and delay a later hard denial

Caps are configuration supplied with an accepted decision. They are only effective when the caller or a mandatory host boundary applies them.

## Tool list precedence

The tool filtering logic follows a clear precedence:

1. If `tool_allowlist` is non-empty, only those tools are allowed (denylist is ignored)
2. If `tool_allowlist` is empty but `tool_denylist` is non-empty, all tools except those in the denylist are allowed
3. If both are empty or absent, no tool restrictions apply

Tool names are case-sensitive and match the `Action.name` field exactly.

## How caps relate to degradation paths

Caps are the protocol-level mechanism behind the degradation strategies described in the Cycles documentation:

- **Downgrade** — reduce model size or generation length → `max_tokens`
- **Disable** — remove expensive tools → `tool_allowlist` / `tool_denylist`
- **Defer** — slow down execution → `cooldown_ms`
- **Limit scope** — bound remaining steps → `max_steps_remaining`

Caps make these degradation paths concrete and server-driven rather than hardcoded in the client.

## Summary

The three-way decision model — ALLOW, ALLOW_WITH_CAPS, DENY — gives Cycles a richer control surface than binary allow/deny.

Caps provide server-driven guidance that lets clients:

- reduce token consumption
- limit remaining steps
- restrict tool usage
- pace execution

This supports a configured degradation path between full execution and hard denial.

## Next steps

To explore the Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)



# Commit Overage Policies in Cycles: REJECT, ALLOW_IF_AVAILABLE, and ALLOW_WITH_OVERDRAFT

::: warning Version note
In v0.1.24, the hardcoded fallback default changed from `REJECT` to `ALLOW_IF_AVAILABLE`. If you are running a server version prior to v0.1.24 and do not set an explicit overage policy on your requests or tenant configuration, the server defaults to `REJECT`. See the [changelog](/changelog) for details.
:::

When actual usage differs from the reserved estimate, the system needs a policy for what to do.

If actual is less than reserved, the unused remainder is released automatically. That is straightforward.

But if actual is more than reserved, the system has a decision to make.

That is what commit overage policies control.

## Quick reference: which policy to use

| Policy | Best for | Commits rejected? | Creates debt? | Tradeoff |
|---|---|---|---|---|
| **REJECT** | Well-estimated actions where hard budget stops are acceptable | Yes, if actual > reserved | No | Strictest enforcement, but rejected commits leave unaccounted gaps |
| **ALLOW_IF_AVAILABLE** (default) | Most workloads — variable-cost actions, LLM calls, tool invocations | Never | No | Always records usage, caps charge to available budget, blocks future reservations when exhausted |
| **ALLOW_WITH_OVERDRAFT** | Must-record actions — external imports, side-effecting operations, SLA-critical work | Only if debt would exceed overdraft limit | Yes | Most accurate ledger, requires operator debt reconciliation |

**Common patterns:**
- **LLM completions with unpredictable token counts** → ALLOW_IF_AVAILABLE
- **Tool calls with known fixed costs** → REJECT (with buffer) or ALLOW_IF_AVAILABLE
- **External API usage imports** → ALLOW_WITH_OVERDRAFT (work already happened)
- **Multi-tenant platform default** → ALLOW_IF_AVAILABLE (safe default, no debt)
- **Strict per-agent budget caps** → REJECT (hard stop when budget is gone)

For a scenario-driven guide, see [Choosing the Right Overage Policy](/how-to/choosing-the-right-overage-policy).

## The three policies

Cycles defines three overage policies, set at reservation time (or on events):

- **REJECT** — refuse the commit if actual exceeds reserved
- **ALLOW_IF_AVAILABLE** — allow if remaining budget can cover the difference
- **ALLOW_WITH_OVERDRAFT** — allow and create debt if necessary

Each policy makes a different tradeoff between ledger accuracy and budget strictness.

### Resolution order

When a reservation or event is created, the server resolves the overage policy in this order:

1. **Request-level** `overage_policy` — if the client specifies one, it is used
2. **Tenant default** `default_commit_overage_policy` — if the tenant has one configured via the Admin API
3. **Hardcoded fallback** — `ALLOW_IF_AVAILABLE`

This means tenant administrators can set an org-wide default (e.g. `REJECT`) and individual requests can still override it.

### Step 0: the commit must be valid

Overage policies only come into play after the commit passes basic validation. Before any policy branch is evaluated, the server checks that:

1. The reservation exists and is owned by the effective tenant — otherwise `404 NOT_FOUND` or `403 FORBIDDEN`
2. The reservation is still committable — not already `COMMITTED` or `RELEASED` (`409 RESERVATION_FINALIZED`) and not past `expires_at_ms + grace_period_ms` (`410 RESERVATION_EXPIRED`)
3. The `actual.unit` matches the reservation's `estimate.unit` — otherwise `400 UNIT_MISMATCH`

A commit that fails any of these checks is rejected regardless of overage policy. Even ALLOW_IF_AVAILABLE, which never rejects a commit on budget grounds, does not accept an invalid commit.

## REJECT

REJECT is the strictest overage policy. Tenant administrators can set it as the default for all reservations and events in their tenant by setting `default_commit_overage_policy` via the Admin API.

REJECT is the simplest and strictest policy.

If `actual > reserved`, the commit is rejected with `409 BUDGET_EXCEEDED`.

### When REJECT is right

- the system can predict costs well enough to reserve sufficient room
- hard budget enforcement is more important than ledger completeness
- a 10–20% buffer on estimates is acceptable
- rejected commits will be retried with a new reservation

### The risk with REJECT

If the action already happened — the model call returned, the tool executed — rejecting the commit creates an unaccounted gap.

The work occurred but is not recorded in the budget ledger. Budget appears more available than it really is.

This is why the spec recommends adding a 10–20% buffer to estimates when using REJECT.

### Practical guidance

- Use REJECT when estimates are reliable
- Add estimation buffers to avoid frequent rejections
- Monitor how often commits are rejected — high rejection rates signal estimation problems

## ALLOW_IF_AVAILABLE (default)

ALLOW_IF_AVAILABLE is the default overage policy when neither the request nor the tenant configuration specifies one. It ensures commits always succeed — the action already happened, so the ledger must reflect it as accurately as possible.

The server checks whether remaining budget across all affected scopes can cover the full delta between actual and reserved.

If yes: the commit succeeds and the full delta is charged atomically.

If no: the commit still succeeds, but the delta is **capped** to the minimum available remaining across all affected scopes (floor 0). The charge is `estimate + capped_delta`. Scopes where the full delta could not be covered are marked `is_over_limit=true`, blocking future reservations until reconciled.

### When ALLOW_IF_AVAILABLE is right

- the system wants to allow modest overages when budget permits
- strict estimation is difficult
- the system should never create debt
- budget accuracy matters but some flexibility is acceptable
- commits should never be rejected after the action has happened

### How it works

**Full delta available:** Suppose a reservation held 100 units and actual usage was 130 units. The delta is 30 units. The server checks whether all affected scopes have at least 30 units of remaining budget. If yes: commit succeeds, 130 is charged.

**Capped delta:** Suppose budget remaining is 200, estimate is 200, actual is 201. The delta is 1 unit. After reservation, remaining is 0. The server caps the delta to 0 (nothing available). Charge is `200 + 0 = 200`. The scope is marked `is_over_limit=true`, blocking future reservations. Budget after: remaining=0, spent includes 200, no debt.

### The key properties

ALLOW_IF_AVAILABLE never creates debt. It never rejects a commit. It charges the maximum amount possible without creating debt, and blocks future reservations when the full overage could not be covered.

This makes it a safe default — work is always accounted for, and the system self-limits when budget is exhausted.

## ALLOW_WITH_OVERDRAFT

ALLOW_WITH_OVERDRAFT allows the commit to succeed even when remaining budget cannot cover the delta.

Instead, the overage is recorded as debt against the scope.

### When ALLOW_WITH_OVERDRAFT is right

- the action has already happened and the ledger must reflect it
- concurrent execution makes strict pre-commit enforcement impractical
- accurate accounting is more important than strict budget boundaries
- the team has processes to reconcile debt afterward

### How it works

Suppose a reservation held 100 units and actual usage was 150 units.

Remaining budget on the scope is only 20 units.

With ALLOW_WITH_OVERDRAFT:

1. The server checks: `(current_debt + delta) <= overdraft_limit`
2. If yes: the commit succeeds, the delta becomes debt, remaining goes negative
3. If no: the commit is rejected with `409 OVERDRAFT_LIMIT_EXCEEDED`

### The overdraft limit

The `overdraft_limit` is set per scope (outside the v0 protocol — typically via admin/operator configuration).

It defines the maximum debt a scope can carry.

If no overdraft_limit is set (or it is zero), ALLOW_WITH_OVERDRAFT behaves like ALLOW_IF_AVAILABLE.

## How to choose

A simple decision framework:

### Use REJECT when:
- estimates are reliable
- you prefer hard stops over partial accounting
- you can add estimation buffers
- you have retry logic for rejected commits

### Use ALLOW_IF_AVAILABLE when:
- estimates sometimes miss
- you want flexibility without debt
- remaining budget is the natural boundary
- you do not want to set up overdraft monitoring

### Use ALLOW_WITH_OVERDRAFT when:
- the work has already happened and must be accounted for
- concurrent execution makes exact pre-commitment impractical
- ledger accuracy is a hard requirement
- you have operator processes for debt reconciliation

## Overage policies on events

Events (`POST /v1/events`) also support all three overage policies with the same semantics.

Since events do not have a preceding reservation, the "overage" is simply whether the event amount exceeds available budget for the scope.

- REJECT: event is rejected if budget is insufficient
- ALLOW_IF_AVAILABLE: event is applied if budget can cover it; if the amount exceeds remaining budget, the charge is capped and the response includes a `charged` field showing the effective amount
- ALLOW_WITH_OVERDRAFT: event is applied with debt if necessary

## Overage policies and concurrency

REJECT and ALLOW_IF_AVAILABLE are atomic per commit.

ALLOW_WITH_OVERDRAFT is also atomic per individual commit, but the overdraft limit check is not atomic across concurrent commits. Multiple concurrent commits may each individually pass the check but collectively push debt past the limit.

This is by design. All commits represent work that already happened. The scope enters over-limit state and blocks future reservations until debt is repaid.

## A practical rollout

A safe rollout path for overage policies:

### Phase 1: Start with REJECT

Use REJECT with 15–20% estimation buffers. Monitor commit rejection rates.

### Phase 2: Move to ALLOW_IF_AVAILABLE for high-variance actions

For actions where estimation is difficult, switch to ALLOW_IF_AVAILABLE. This reduces rejected commits while maintaining budget boundaries.

### Phase 3: Use ALLOW_WITH_OVERDRAFT for must-record actions

For actions where the work has already happened (external API calls, side-effecting operations), use ALLOW_WITH_OVERDRAFT. Set up overdraft monitoring and debt reconciliation processes.

## Summary

The three commit overage policies give teams control over the tradeoff between budget strictness and ledger accuracy:

- **REJECT** — strictest, may create unaccounted gaps
- **ALLOW_IF_AVAILABLE** (default) — flexible, no debt, always commits, caps overage to available
- **ALLOW_WITH_OVERDRAFT** — most accurate, creates debt, requires reconciliation

Most systems benefit from using different policies for different action classes: REJECT for well-estimated actions, ALLOW_IF_AVAILABLE for variable ones, and ALLOW_WITH_OVERDRAFT for must-record side effects.

## Next steps

To explore the Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)



# Correlation and Tracing in Cycles

Cycles produces data on four planes — runtime responses, webhook deliveries, audit-log entries, and emitted events. Correlation identifiers stitch those planes into a single causal picture.

The current YAML specifications define a W3C Trace Context-compatible correlation contract across the server suite. The reference services implement request and trace propagation; server-composed admin `correlation_id` values are also implemented. One current gap is called out below: the reference runtime does not yet populate the protocol's deterministic event-cluster `correlation_id`.

## The three-tier model

Cycles carries three correlation identifiers, each with a different grain.

| Identifier | Grain | Lifetime |
|---|---|---|
| `request_id` | One HTTP request | Per-request |
| `trace_id` | One logical operation (may span many requests) | Per-operation |
| `correlation_id` | An event-stream cluster or admin operation fan-out | Server-set: deterministic hash required by the runtime protocol, or an implemented operation ID for selected admin operations |

- **`request_id`** is server-generated for every inbound HTTP request. It appears in every error response, audit-log entry, and event that is causally downstream of that request. Use it to correlate the side effects of one specific HTTP call.
- **`trace_id`** identifies a logical operation that may cross several HTTP boundaries (for example: a client's reserve → provider call → commit) when the caller propagates it. It is a 32-hex-character W3C Trace Context-compatible identifier. Use it to join the Cycles records that carry the field and application telemetry for the rest.
- **`correlation_id`** is server-managed and scoped to event rows; callers do not set it through runtime request bodies or tracing headers. The runtime YAML requires a deterministic hash over `(tenant_id, scope, action_kind_or_risk_class, window, window_key)` for event-stream clusters, but the current reference runtime passes `null` on its implemented emit paths, so consumers must not depend on that join yet. The current admin server does populate explicit operation IDs for selected lifecycle, bulk, webhook, and tenant-close operations, such as `webhook_create:<id>` and `webhook_bulk_action:<action>:<request_id>`. See the [Event Payloads Reference](/protocol/event-payloads-reference) for implemented shapes.

::: warning Don't confuse with `metadata.trace_id`
[Standard Metrics and Metadata](/protocol/standard-metrics-and-metadata-in-cycles) documents application-level correlation keys that callers can put in the `metadata` map on commits and events — free-form strings the server stores but does not interpret. Name them distinctly (e.g. `external_trace_id`, `app_request_id`) rather than reusing `trace_id`/`request_id`, which are the server-managed identifiers described on this page (32-hex W3C, flowing on response headers, error bodies, events, audit rows, and webhook deliveries). The two coexist: your `metadata.external_trace_id` joins Cycles data with your own distributed tracing, while the server `trace_id` joins across Cycles planes.
:::

::: warning Propagation does not add fields to reservation records
The caller must send the same trace context on each related HTTP request; independently server-generated trace IDs are per request. The current `Reservation` model does not persist `trace_id`, and successful reserve, commit, release, and extend calls do not each emit lifecycle events. Keep the reservation ID and trace ID together in application logs when you need an end-to-end reserve/provider/settlement reconstruction.
:::

## Inbound header precedence

For every inbound HTTP request, the server derives `trace_id` in this strict order:

1. **`traceparent` header** — W3C Trace Context v00. Adopted when present AND well-formed.
2. **`X-Cycles-Trace-Id` header** — 32 lowercase hex characters (`^[0-9a-f]{32}$`). Used only when there is no valid `traceparent`.
3. **Server-generated** — 16 random bytes encoded as 32 lowercase hex, when neither header is present or well-formed. All-zero is invalid per W3C §3.2.2.3 and is re-rolled.

**Malformed tolerance.** A malformed correlation header MUST NOT cause the request to be rejected. The server silently falls through to the next rule. A misbehaving upstream proxy cannot break the API.

**Disagreement.** If both `traceparent` and `X-Cycles-Trace-Id` are present, valid, and disagree, `traceparent` wins.

## Outbound propagation

### On HTTP responses

Every response on every plane (`2xx`, `4xx`, `5xx`) carries:

```http
X-Cycles-Trace-Id: <32-hex-lowercase>
```

In addition, `ErrorResponse` bodies carry an optional `trace_id` field:

```json
{
  "error": "BUDGET_EXCEEDED",
  "message": "Insufficient budget in scope tenant:acme",
  "request_id": "req-abc-123",
  "trace_id": "0af7651916cd43dd8448eb211c80319c"
}
```

The `trace_id` field is OPTIONAL on the schema. Conformant v0.1.25.14+ runtime servers and v0.1.25.31+ admin servers populate it on every error response.

### On webhook deliveries

Every webhook delivery emitted by the events service carries two cross-surface headers on top of the existing delivery headers:

```http
X-Cycles-Trace-Id: <32-hex-lowercase>
traceparent: 00-<trace_id>-<16-hex-span>-<trace-flags>
```

- `X-Cycles-Trace-Id` — always present. Matches `X-Cycles-Trace-Id` on the originating response.
- `traceparent` — always present. W3C Trace Context v00. The `span-id` is freshly generated per outbound delivery (NOT reused from the inbound request). The `trace-flags` byte preserves the inbound W3C `traceparent` sampling decision when one was present; otherwise defaults to `01` (sampled).

These two are the normative delivery headers per the spec's required-header list. The reference implementation additionally sends an `X-Request-Id` header when the originating event carries a `request_id`, but that header is not part of the spec's required set — the normative carrier for `request_id` is the event envelope body.

### Inside emitted events

Standard event payloads carry:

| Field | Contract |
|---|---|
| `request_id` | Populated on every event causally downstream of an HTTP request — including async and queued work that spans thread / process boundaries. Pre-v0.1.25 events may lack it. |
| `trace_id` | OPTIONAL on the schema; populated by conformant v0.1.25.14+ runtime servers. |
| `correlation_id` | Server-managed. The runtime YAML requires deterministic event clusters, but the current reference runtime leaves this field absent on implemented runtime emits. Selected admin operations populate explicit IDs such as `webhook_create:<id>`, bulk-action IDs, and cascade IDs. |

### Inside audit-log entries

Every `AuditLogEntry` carries `request_id` and (OPTIONAL) `trace_id`. These fields flow from the inbound request onto the audit row at write time. Admin-driven operations tagged with metadata such as `metadata.actor_type=admin_on_behalf_of` or `metadata.actor_type=admin` carry the same `trace_id` as the request that triggered them.

### Inside webhook delivery records

The `WebhookDelivery` schema carries three OPTIONAL fields as of governance-admin spec v0.1.25.28:

| Field | Purpose |
|---|---|
| `trace_id` | Captured at dispatch time from the originating event. Used by the events service to construct outbound `X-Cycles-Trace-Id` and `traceparent` headers. |
| `trace_flags` | W3C `trace-flags` byte (2 hex chars) to use when building the outbound `traceparent`. Preserves the inbound sampling decision. |
| `traceparent_inbound_valid` | Whether the originating HTTP request presented a valid W3C `traceparent`. When `true`, the dispatcher honors `trace_flags`; when `false` or null, it defaults to `01` (sampled). |

## Cross-plane propagation

`trace_id` travels:

- **Inbound request → response header** — echoed in `X-Cycles-Trace-Id` on every HTTP response.
- **Request → audit-log entry** — written for admin operations and the runtime's admin-on-behalf-of release path that create an audit row. Ordinary tenant reserve/commit/release calls do not each create an `AuditLogEntry`.
- **Request → emitted events** — attached to every event that is a side effect of the request, including events emitted from async workers (`ReservationExpiryService` for example mints a fresh `trace_id` per sweep batch so all `reservation.expired` events in that batch correlate to each other).
- **Events → webhook deliveries** — carried through to each outbound HTTP POST as `X-Cycles-Trace-Id` and embedded in `traceparent`.
- **Across thread / queue / process boundaries** — REQUIRED when work is causally downstream of a request. Internally originated sweepers may mint their own context; the reference reservation-expiry sweeper creates one trace ID per batch.

## Querying by correlation identifiers

The admin plane supports exact-match filters on correlation identifiers:

### `GET /v1/admin/events`

| Query parameter | Effect |
|---|---|
| `correlation_id=<id>` | Narrows to events carrying one implemented server-composed correlation ID. Most useful for admin bulk actions, webhook lifecycle operations, and tenant-close cascades; current runtime events generally lack it. |
| `trace_id=<32-hex>` | Narrows to events emitted during one logical operation. May span multiple requests. |
| `request_id=<id>` | Narrows to events that are side effects of one specific HTTP request. |

### `GET /v1/admin/audit/logs`

| Query parameter | Effect |
|---|---|
| `trace_id=<32-hex>` | Narrows to audit rows for one logical operation. |
| `request_id=<id>` | Narrows to audit rows for one specific HTTP request. |

These are the only two admin endpoints with server-side request/trace filters; only the events endpoint also accepts `correlation_id`. The webhook-delivery list endpoints (`GET /v1/admin/webhooks/{subscription_id}/deliveries` and the tenant-scoped variant) accept only `status` / `from` / `to` / pagination parameters — there is no `trace_id` query parameter. To join deliveries into a trace, filter client-side on the `trace_id` field of each `WebhookDelivery` record.

Both filters are post-hydration predicates applied null-safely — entries with null field values (historical writes, off-request emissions, internal sweeper work) cannot satisfy a supplied filter value. Pre-v0.1.25.14 runtime entries and pre-v0.1.25.31 admin entries may lack `trace_id` and silently drop out of these joins; use `request_id` for those (the `request_id` contract predates `trace_id`).

### Phased-rollout tolerance

The four servers don't have to upgrade in lockstep. During a phased rollout, every combination is wire-compatible — but the `trace_id` filter will return partial results until every plane is at the minimum version:

| Plane | Minimum for `trace_id` population |
|---|---|
| Runtime (`cycles-server`) | v0.1.25.14 |
| Admin (`cycles-server-admin`) | v0.1.25.31 |
| Events (`cycles-server-events`) | v0.1.25.7 |
| Dashboard (`cycles-dashboard`) | v0.1.25.39 (consumes the admin filter) |

Concrete effect: if runtime is at v0.1.25.15 (writes `trace_id` on events) but admin is still at v0.1.25.26 (does NOT yet persist `trace_id` on `WebhookDelivery` records), webhook deliveries for that window will lack `trace_id` in admin queries even though the underlying event has one. Upgrade admin to v0.1.25.31+ to close the gap. Events v0.1.25.8 proactively back-fills `trace_id` onto `Delivery` records from the originating `Event.trace_id` during this window as a best-effort safety net.

If you can't upgrade every plane at once, pin `trace_id`-based alerting to queries that tolerate partial coverage (e.g., show results even when some webhook deliveries lack the field).

## A practical join

Given a single failing request, an operator can reconstruct the entire operation by reading the `trace_id` out of the error response, then walking the two admin endpoints that accept a `trace_id` filter — and filtering the delivery list client-side:

```bash
TID=0af7651916cd43dd8448eb211c80319c

# 1. The request reached the server. What decision was recorded?
curl -s "http://localhost:7979/v1/admin/audit/logs?trace_id=$TID" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  | jq '.logs[] | {operation, status, error_code, metadata}'

# 2. What events were emitted as side effects?
curl -s "http://localhost:7979/v1/admin/events?trace_id=$TID" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  | jq '.events[] | {event_type, data}'

# 3. What webhook deliveries went out as a consequence?
#    (no trace_id query parameter here — filter client-side on the record's trace_id field)
curl -s "http://localhost:7979/v1/admin/webhooks/<subscription-id>/deliveries" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  | jq --arg tid "$TID" '.deliveries[] | select(.trace_id == $tid) | {status, response_status, trace_flags}'
```

Three calls, one trace ID — two server-side filters plus one client-side filter — return the records that each plane retained for that trace. Application authorization and external execution outcomes still come from your own trace-linked logs.

## Logging `trace_id` in client code

Conformant client logging captures both `request_id` and `trace_id` from the error response. When should you use which?

- **`request_id`** is enough when you're debugging a single failed call — it points at one audit row and its immediate side effects.
- **`trace_id`** is the right choice when the symptom may involve retries, async commits, admin-on-behalf-of releases, or cross-plane fan-out. It narrows to one logical operation regardless of how many HTTP hops happened.

Log both. `request_id` is the tightest predicate; `trace_id` is the widest useful one.

```python
try:
    result = summarize(text)
except CyclesProtocolError as e:
    # SDK field (present on v0.1.25-aware SDKs) with response-header fallback.
    trace_id = getattr(e, "trace_id", None) or (
        e.response_headers.get("X-Cycles-Trace-Id") if hasattr(e, "response_headers") else None
    )
    logger.error(
        "cycles error",
        extra={
            "error_code": e.error_code,
            "status": e.status,
            "request_id": e.request_id,
            "trace_id": trace_id,
        },
    )
    raise
```

The SDKs expose `trace_id` on the error object where available. Older SDK versions that predate v0.1.25 support return `None` from `e.trace_id`; the fallback above pulls from the `X-Cycles-Trace-Id` response header instead. That header is always present on v0.1.25.14+ runtime and v0.1.25.31+ admin responses, so the fallback covers every case where the server populates the field.

## Backward compatibility

The correlation contract is purely additive:

- No new REQUIRED fields on existing schemas.
- Old clients silently ignore the new `X-Cycles-Trace-Id` response header.
- Old webhook subscribers silently ignore the new outbound `X-Cycles-Trace-Id` and `traceparent` headers (and the reference implementation's `X-Request-Id`).
- Servers that predate the contract (runtime below v0.1.25.14, admin below v0.1.25.31, events below v0.1.25.7) remain wire-compatible: `trace_id` is an OPTIONAL property, and `ErrorResponse.additionalProperties: false` is preserved because `trace_id` is a DECLARED property, not an undeclared extra.

## Next steps

- [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol) — the outbound headers in context
- [Error Codes and Error Handling](/protocol/error-codes-and-error-handling-in-cycles) — `trace_id` in error responses
- [Searching and Sorting Admin List Endpoints](/how-to/searching-and-sorting-admin-list-endpoints) — the audit-log filter DSL, including `trace_id` and `request_id` filters
- [Force-Releasing Stuck Reservations](/how-to/force-releasing-stuck-reservations-as-an-operator) — uses `trace_id` to audit admin-on-behalf-of releases



# CyclesEvidence Envelopes in Cycles

This page is the protocol reference for CyclesEvidence — the signed, content-addressed audit envelope behind the [verifiable-audit concept](/concepts/cycles-evidence-verifiable-audit-for-agent-decisions). For *why* it exists, start there; this page is the *how*.

The consumer surface (`cycles_evidence` on responses, `GET /v1/evidence/{id}`, and `GET /v1/.well-known/cycles-jwks.json`) is defined in `cycles-protocol-v0.yaml`. The envelope and signer-authority rules are specified in [`cycles-evidence-v0.2.yaml`](https://github.com/runcycles/cycles-protocol/blob/main/cycles-evidence-v0.2.yaml). The wire `schema_version` remains `cycles-evidence/v0.1` for compatibility; v0.2 adds the normative JWK Set authority layer around that envelope shape.

## The `cycles_evidence` reference

Every decide / reserve / commit / release response — and budget/lifecycle **denial** responses — may carry an optional `cycles_evidence`:

```json
"cycles_evidence": {
  "evidence_id": "8403bed4…7030",
  "cycles_evidence_url": "https://cycles.example.com/v1/evidence/8403bed4…7030"
}
```

- `evidence_id` — lowercase 64-hex SHA-256, the content address of the signed envelope.
- `cycles_evidence_url` — `{server_id}/evidence/{evidence_id}`. `server_id` already includes the `/v1` base, so the join adds only `/evidence/{id}`.

It is **transport metadata, not attested** — present for the caller's convenience and computed over the response *without* this field (see [Non-self-referential](#non-self-referential) below). It is absent when evidence emission is disabled, or for errors raised before a decision was reached (validation/auth failures). Additive and `@JsonInclude(NON_NULL)`: a client that ignores it is unaffected.

## The envelope and its five artifact types

`GET /v1/evidence/{id}` returns the signed envelope verbatim:

```json
{
  "schema_version": "cycles-evidence/v0.1",
  "artifact_type": "reserve",
  "server_id": "https://cycles.example.com/v1",
  "signer_did": "b10554…c522",
  "issued_at_ms": 1781436904050,
  "trace_id": "b2a0ab88…dc02",
  "payload": { "reserve": { "request": { … }, "response": { … } } },
  "evidence_id": "8403bed4…7030",
  "signature": "4bc8cb9a…8c08"
}
```

| `artifact_type` | Endpoint | Payload |
|---|---|---|
| `decide` | `POST /v1/decide` | `{ request, response }` |
| `reserve` | `POST /v1/reservations` | `{ request, response }` |
| `commit` | `POST /v1/reservations/{id}/commit` | `{ reservation_id, request, response }` |
| `release` | `POST /v1/reservations/{id}/release` | `{ reservation_id, request, response }` |
| `error` | any of the above (4xx/5xx) | `{ endpoint, http_status, [reservation_id], [request], response }` |

`commit` / `release` (and commit/release `error`s) **hoist `reservation_id`** into the payload so an evidence-only reader can reconstruct the authorization → settlement chain without the URL.

### Denials → the `error` artifact

A non-dry `reserve` over budget is **not** a `200` with `decision: DENY` — it is an `HTTP 409` with `error: BUDGET_EXCEEDED`, captured as an `error` envelope (`endpoint: "POST /v1/reservations"`, `http_status: 409`). The other post-evaluation budget/lifecycle denials behave the same — `BUDGET_FROZEN`, `BUDGET_CLOSED`, `OVERDRAFT_LIMIT_EXCEEDED`, `DEBT_OUTSTANDING`, `TENANT_CLOSED`, `UNIT_MISMATCH`, and the commit/release terminal-state denials `RESERVATION_FINALIZED` (409) and `RESERVATION_EXPIRED` (410). Pre-evaluation failures (validation, auth, malformed body) carry **no** `cycles_evidence` — no decision was reached, so there is nothing to attest. (A dry-run preflight denial, by contrast, is a `200` captured as `reserve` evidence — it is the canonical "would this be allowed?" attestation.)

## `evidence_id` — the content-hash recipe (normative)

1. Build the envelope with every field populated **except** `evidence_id` and `signature`, both set to the empty string `""`.
2. Canonicalize per RFC 8785 (JCS); UTF-8 encode.
3. `evidence_id` = lowercase hex SHA-256 of those bytes.

Because the id is a pure function of the contents (no private key), **Cycles computes it synchronously and returns it on the response**, even though signing happens later.

## Signature derivation (normative)

1. Take the envelope with `evidence_id` now populated and `signature` still `""`.
2. Canonicalize again (JCS), UTF-8 encode.
3. `signature` = lowercase hex of the Ed25519 signature over those bytes, using the server's signing key (named by `signer_did`).

This is the same id-then-signature ordering used elsewhere in the agent-trust ecosystem, so a consumer that can verify one of those receipts can verify a CyclesEvidence envelope with the same primitives.

## Non-self-referential

The `cycles_evidence` ref is stamped onto the response **after** `evidence_id` is computed. So the `payload.<artifact>.response` inside the envelope never contains `cycles_evidence` — the content hash is never self-referential. The response mirrors in the normative v0.2 spec keep `additionalProperties: false` and omit the ref to make this explicit.

## How to verify

Given an envelope:

1. **Re-derive `evidence_id`** per the recipe above and compare byte-for-byte. Mismatch ⇒ tampered or canonicalization error.
2. **Resolve signer authority** by fetching `GET {server_id}/.well-known/cycles-jwks.json` and selecting the Ed25519 JWK whose `[cycles_nbf_ms, cycles_exp_ms)` window covers the envelope's `issued_at_ms`. The selected key's public bytes must match `signer_did`.
3. **Verify the Ed25519 `signature`** (with `evidence_id` populated, `signature` emptied) against that public key.
4. **Check the `artifact_type` ↔ `payload` pairing** (e.g. `artifact_type: commit` requires `payload.commit`).

Signature *validity* proves the envelope was signed by the key in `signer_did`. Signer *authority* proves that key was published by the issuing `server_id` for the envelope's issuance window. The JWK Set is the normative v0.2 authority layer. If a server does not publish JWKS, consumers can still run in a pinned-signer (`binding_only`) posture by comparing `signer_did` to an expected signer out of band. Why validity and authority are different questions: [A Valid Signature Doesn't Tell You Who Signed It](/blog/a-valid-signature-doesnt-tell-you-who-signed-it).

### Verification dispositions

A conformant verifier reports **exactly one** of five dispositions, keeping the two axes distinct:

- `authentic` — signature valid **and** authority established: the verifying key is the one window-covering key selected deterministically from `server_id`'s JWK Set.
- `binding_only` — signature cryptographically valid, but no JWKS authority was resolved (raw-hex with no set lookup, or a pin-only deployment). The companion `signer_pin_matched` boolean says whether an `expected_signer` pin was present and matched `signer_did`.
- `signer_authority_failed` — resolution *succeeded* (the JWK Set was fetched and parsed) but the key is not authorized for this envelope: DID↔`server_id` hash mismatch, no window-covering key, the raw-hex key absent from the set, or an ambiguous / duplicate-`kid` selection. Neither a network failure nor a forgery.
- `signer_resolution_failed` — the JWK Set could not be retrieved or parsed (network, 404/transient, unparseable body). About *obtaining* the set, not searching it; establishes nothing about the bytes and must never be reported as `signature_invalid`.
- `signature_invalid` — the bytes do not verify against the resolved/named key (tamper).

## Signer-key resolution and rotation

### The two `signer_did` forms

`signer_did` names the Ed25519 signer in one of two forms:

1. **Raw hex** — the 32-byte public key as 64 lowercase hex chars. Self-describing: a verifier checks the `signature` against it directly, with no resolution (the only form v0.1 requires). Authority then means "a window-covering JWK whose `x` decodes to the same 32 bytes exists in `server_id`'s set" — no `kid` needed (a JWK's `x` is `base64url(pubkey)`, the raw-hex form is `hex(pubkey)`; same bytes).
2. **`did:cycles:<server_id_hash>#<kid>`** — `<server_id_hash>` is lowercase `hex(sha256(server_id))`, binding the DID to the envelope's `server_id`; the `#<kid>` fragment names the key within the server's JWK Set (it equals that JWK's `kid`). A verifier resolves this form to establish signer *authority*, not just signature validity. A DID whose `<server_id_hash>` doesn't equal `hex(sha256(server_id))` fails authority — the DID does not bind to this envelope's `server_id`.

Step 2 needs the right public key. A server publishes its keys as a JWK Set:

```
GET {server_id}/.well-known/cycles-jwks.json     # operationId: getEvidenceJwks
```

Public and unauthenticated (it carries public keys only). Each entry is an Ed25519 OKP JWK with a validity window:

```json
{
  "kty": "OKP", "crv": "Ed25519", "alg": "EdDSA",
  "x": "<base64url of the 32 raw public-key bytes>",
  "kid": "2026-h2",
  "cycles_nbf_ms": 1781000000000,
  "cycles_exp_ms": 1796000000000,
  "status": "retired"
}
```

The **active** key omits `cycles_exp_ms` (open-ended) and has `status: active`. A server not doing signer-key resolution publishes nothing — the endpoint `404`s, and consumers stay on the pinned-signer (`binding_only`) path.

**Window-gated selection.** A verifier selects the key whose `[cycles_nbf_ms, cycles_exp_ms)` window covers the envelope's `issued_at_ms` — never "the current key." So an envelope signed two rotations ago still verifies against the key that was valid when it was signed; the set keeps **retired** keys for exactly this. `status` is advisory — selection is by window. (The forgery this prevents: [Rotating Keys Shouldn't Rewrite History](/blog/rotating-keys-shouldnt-rewrite-history).)

**Exactly one candidate.** Selection is a total function with no implementation discretion: the window gate must be satisfied by **exactly one** candidate key. Zero covering keys, overlapping windows that leave two candidates, a duplicate `kid` in the set, a `did:cycles` fragment with no matching `kid`, or (raw-hex) zero or multiple window-covering JWKs matching the key bytes — all are `signer_authority_failed`, never a silent pick.

### Rotating the signing key (operator procedure)

The windows must tile without overlapping. On rotation:

1. Generate the new Ed25519 key pair and deploy the private key only to `cycles-server-events` as `EVIDENCE_SIGNING_PRIVATE_KEY_HEX`.
2. Make the new public key active on the runtime server — `EVIDENCE_SIGNING_SIGNER_DID` = the new raw-hex public key, `EVIDENCE_SIGNING_KID` = the active JWK `kid`, and `EVIDENCE_SIGNING_NBF_MS` = the rotation time (epoch ms).
3. Deploy the same public `EVIDENCE_SIGNING_SIGNER_DID` and `EVIDENCE_SERVER_ID` to `cycles-server-events` so the worker signs envelopes with the same identity the runtime publishes and used when computing `evidence_id`.
4. Append the old public key to the runtime server's `EVIDENCE_SIGNING_RETIRED_KEYS` — a JSON array of `{"signer_did","kid","nbf_ms","exp_ms"}` — with `exp_ms` = that same rotation time.

The retiring key's window then ends exactly where the new key's begins.

**Fail-safe, never fail-closed.** If `nbf-ms` is left below the latest retired `exp_ms`, the published active window is **clamped up** to that boundary (with a warning), so the current key is never published as authoritative for pre-rotation `issued_at_ms` by accident. A retired entry that can't be published (malformed hex, empty/inverted window, out-of-range bound, duplicate `kid`) is dropped, not fatal; if the whole `retired-keys` value is unusable, the server logs an error and keeps serving the active key — it never refuses to publish, which would break verification of *all* current evidence.

## Producer / signer split

- **`cycles-server`** computes `evidence_id` synchronously, returns `cycles_evidence`, serves `GET /v1/evidence/{id}` and `GET /v1/.well-known/cycles-jwks.json`, and holds only the **public** identity.
- **`cycles-server-events`** asynchronously builds, **Ed25519-signs** (the private key lives only here), and stores the envelope content-addressed. It recomputes the id and **dead-letters on drift**, so producer/signer config mismatch fails closed.

Because signing is async, a fetch immediately after the response may return a transient `404` — treat it as not-yet-available and retry.

## Enabling it

Evidence is **off until a shared signing identity is configured**:

- `EVIDENCE_SERVER_ID` on both services — the issuer base URL, including `/v1`, used in evidence URLs and envelopes.
- `EVIDENCE_SIGNING_SIGNER_DID` on both services — the raw-hex public Ed25519 key.
- `EVIDENCE_SIGNING_PRIVATE_KEY_HEX` only on `cycles-server-events` — the raw-hex private Ed25519 key.
- `EVIDENCE_SIGNING_KID`, `EVIDENCE_SIGNING_NBF_MS`, and `EVIDENCE_SIGNING_RETIRED_KEYS` only on `cycles-server` — public JWKS metadata and rotation history.

See the operator [identity enablement runbook](https://github.com/runcycles/cycles-server-events/blob/main/docs/evidence-identity-enablement.md).

## Related

- [CyclesEvidence: Verifiable Audit for Agent Decisions](/concepts/cycles-evidence-verifiable-audit-for-agent-decisions) — the why.
- [Error Codes and Error Handling](/protocol/error-codes-and-error-handling-in-cycles) — the denial codes that surface as `error` evidence.
- [Correlation and Tracing](/protocol/correlation-and-tracing-in-cycles) — the `trace_id` carried on every envelope.



# Debt, Overdraft, and the Over-Limit Model in Cycles

Budget enforcement in Cycles is designed to prevent unbounded execution.

But real systems sometimes cross budget boundaries.

A model call may cost more than expected.
A retry may push actual usage past the reserved amount.
Concurrent commits may collectively exceed available budget.

When this happens, the system needs a clear model for what occurred, how much debt exists, and when normal operation can resume.

That is what the debt and overdraft model provides.

## The problem: what happens when actuals exceed budget?

In a strict system, any overage is rejected. The commit fails.

That is the behavior with overage policy `REJECT`. The default policy since v0.1.24 is `ALLOW_IF_AVAILABLE`, which caps the charge to available budget instead of rejecting.

But sometimes the work has already happened. The model call returned. The tool executed. The side effect occurred.

Rejecting the commit at that point does not undo the work. It just creates an unaccounted gap in the ledger.

The overdraft model solves this by allowing the system to record what actually happened, even when budget is insufficient, while clearly marking the resulting debt.

## How overdraft works

When a commit or event uses the `ALLOW_WITH_OVERDRAFT` overage policy and the actual amount exceeds available budget:

1. The server checks whether `(current_debt + delta) <= overdraft_limit` for each affected scope
2. If yes: the commit succeeds, the delta is added to debt, and remaining can go negative
3. If no: the commit is rejected with `409 OVERDRAFT_LIMIT_EXCEEDED`

For commits, the delta is the overage beyond what was reserved. Events (`POST /v1/events`) have no reservation to net against — the full actual amount is the debit, so the overdraft check becomes `(current_debt + actual) <= overdraft_limit` and, on success, the full actual is added to debt.

This means the system can absorb overages up to a defined limit, then stops.

## Key concepts

### debt

Debt represents actual consumption that occurred when insufficient budget was available.

It is a real number in the balance ledger. It is not theoretical. It reflects work that happened and was accounted for, even though budget did not fully cover it.

### overdraft_limit

The overdraft limit defines the maximum debt a scope is allowed to accumulate.

If the limit is 10,000 units, the scope can absorb up to 10,000 units of debt before further overages are rejected.

If the limit is absent or zero, no overdraft is permitted (the scope behaves as if using `ALLOW_IF_AVAILABLE`).

### is_over_limit

When debt exceeds the overdraft limit — typically due to concurrent commits — the scope enters an over-limit state.

The same state can also be entered with **zero debt**: under `ALLOW_IF_AVAILABLE`, when the full overage delta cannot be covered, the server caps the charge and sets `is_over_limit` to `true` — without ever creating debt.

In this state:

- `is_over_limit` is `true` on the balance
- all new reservations against that scope are rejected with `409 OVERDRAFT_LIMIT_EXCEEDED`
- existing active reservations can still be committed or released normally

The scope remains blocked until reconciled — debt repaid below the overdraft limit, or, on the zero-debt path, `is_over_limit` cleared by operator action.

### remaining (can be negative)

In Cycles, the `remaining` balance uses a `SignedAmount` — it can go negative.

A negative remaining means the scope has consumed more than its allocated budget. This is only possible when overdraft is enabled.

The formula is: `remaining = allocated - spent - reserved - debt`

## How concurrent commits create over-limit state

The overdraft limit check is per-commit, not atomic across concurrent commits.

Consider this scenario:

- Scope has overdraft_limit = 5,000
- Current debt = 0
- Two concurrent commits each need 4,000 in overage

Each commit individually checks: `(0 + 4,000) <= 5,000` → passes.

Both commits succeed. But now debt = 8,000, which exceeds overdraft_limit = 5,000.

The scope enters over-limit state.

This is by design. The commits represent work that already happened. Rejecting them would create unaccounted gaps. Instead, the system records the reality and blocks future work until the situation is resolved.

## What happens when a scope is over-limit

When `is_over_limit` is true:

1. **New reservations are blocked.** Any attempt to reserve against the scope returns `409 OVERDRAFT_LIMIT_EXCEEDED`.

2. **Existing reservations can be finalized.** Active reservations can still be committed or released. This prevents in-flight work from being stranded.

3. **Decide returns DENY.** The decide endpoint returns `DENY` with an appropriate reason code instead of a 409 error.

4. **The block is automatic.** No operator action is needed to enforce the block — it is protocol-level.

5. **Recovery is through funding or operator action.** The scope is unblocked when debt is repaid below the overdraft limit through budget funding operations (which are outside the scope of the v0 protocol). When the over-limit state was entered with zero debt (the capped `ALLOW_IF_AVAILABLE` path), there is no debt to repay — an operator clears `is_over_limit` directly.

## Debt vs budget denial

These are different situations:

**Budget denial** (`BUDGET_EXCEEDED`): the request is refused and no debt is created. At commit time this happens only under the `REJECT` overage policy — `ALLOW_IF_AVAILABLE` never rejects a commit; it caps the charge to available remaining instead. At reservation time, a denial occurs when the estimate exceeds remaining budget, regardless of overage policy.

**Debt creation** (`ALLOW_WITH_OVERDRAFT`): the scope has insufficient budget, but the overage policy allows debt. The commit succeeds, debt is recorded, and the ledger reflects reality.

**Over-limit block** (`OVERDRAFT_LIMIT_EXCEEDED`): debt has exceeded the overdraft limit. Future reservations are blocked until debt is repaid.

**Outstanding debt block** (`DEBT_OUTSTANDING`): debt exists and no overdraft limit is configured (overdraft_limit is absent or 0). New reservations are blocked because the scope has unresolved debt with no overdraft tolerance. When an `overdraft_limit > 0` is set, debt within the limit does not block new reservations.

Note: when `is_over_limit` is true, the server returns `OVERDRAFT_LIMIT_EXCEEDED` instead of `DEBT_OUTSTANDING`, even if debt > 0. The over-limit error takes precedence.

## When to use overdraft

Overdraft is most useful when:

- the cost of unaccounted work is worse than the cost of debt
- model calls and tool actions cannot be undone after execution
- the system needs accurate ledger state even under budget pressure
- concurrent execution makes strict pre-commit enforcement impractical
- operators prefer to reconcile debt after the fact rather than lose ledger accuracy

## When not to use overdraft

Overdraft is less appropriate when:

- strict budget enforcement is required (use `REJECT`)
- the system should never exceed allocated budget under any circumstances
- debt reconciliation processes are not in place
- the team prefers hard stops over post-hoc resolution

## Monitoring over-limit state

The spec recommends operators monitor over-limit scopes with:

- **Dashboard** showing scopes where `is_over_limit` is true
- **Warning alerts** when debt reaches 80% of overdraft_limit
- **Critical alerts** when debt exceeds 100% (over-limit state)
- **Metrics** tracking `debt_utilization = debt / overdraft_limit`

The recommended operator runbook:

1. Investigate which commits caused the over-limit state
2. Determine whether the overdraft limit should increase (normal variance) or whether this is anomalous (incident)
3. Fund the scope to repay debt below the limit
4. Monitor that `is_over_limit` returns to false
5. Operations resume automatically

## Summary

The debt and overdraft model in Cycles provides a controlled way to handle budget overages in real production systems.

Instead of choosing between "reject everything" and "allow everything," teams can define an overdraft limit that absorbs reasonable overages while blocking new work when debt becomes excessive.

The key mechanisms:

- **overdraft_limit** defines how much debt a scope can tolerate
- **debt** records actual consumption beyond available budget
- **is_over_limit** blocks new reservations when debt exceeds the limit — or when a capped `ALLOW_IF_AVAILABLE` overage could not be fully covered, even with zero debt
- **recovery** happens through budget funding (or operator clearing on the zero-debt path), which is outside the v0 protocol scope

This gives teams accurate ledger state, bounded risk, and a clear path to resolution.

## Next steps

To explore the Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)



# Dry Run: Shadow Mode Evaluation in Cycles

Before enforcing budget limits in production, teams need a way to test the full reservation path without actually holding budget.

That is what dry run provides.

Setting `dry_run: true` on a reservation request tells the server to evaluate the request as if it were real — including scope derivation, budget checks, decision logic, and cap computation — but without modifying any budget state.

## What dry run does

A dry run reservation request:

1. Evaluates the Subject and derives canonical scopes
2. Checks budget availability across all derived scopes
3. Returns a decision (ALLOW, ALLOW_WITH_CAPS, or DENY)
4. Returns affected_scopes showing which scopes were evaluated
5. Returns caps if the decision is ALLOW_WITH_CAPS
6. Optionally returns balance snapshots for operator visibility

All of this happens without:

- creating a reservation
- modifying any balance
- requiring a subsequent commit or release

## Dry run response rules

The protocol defines specific rules for dry run responses:

### reservation_id and expires_at_ms are absent

A dry run does not create a reservation, so `reservation_id` and `expires_at_ms` MUST be absent from the response (not present with a null value — the fields must not appear).

### affected_scopes is always populated

Regardless of the decision outcome — ALLOW, ALLOW_WITH_CAPS, or DENY — the `affected_scopes` field must be populated.

This is important for debugging. Even when a dry run returns DENY, the client can see which scopes were evaluated and identify where the bottleneck is.

### caps follow the same rules

If the decision is ALLOW_WITH_CAPS, `caps` is present with the same constraints that would apply to a real reservation.

If the decision is ALLOW or DENY, `caps` is absent.

### reason_code on DENY

When a dry run returns DENY, the `reason_code` field should be populated. This is the primary diagnostic signal for understanding why the dry run was denied.

`DecisionReasonCode` is an open string (as of v0.1.25) with seven documented known values: `BUDGET_EXCEEDED`, `BUDGET_FROZEN`, `BUDGET_CLOSED`, `BUDGET_NOT_FOUND`, `OVERDRAFT_LIMIT_EXCEEDED`, `DEBT_OUTSTANDING`, `TENANT_CLOSED` (added in spec v0.1.25.13 — a fresh dry run on a closed owning tenant returns `decision=DENY` with this reason, never `409 TENANT_CLOSED`; the persisting create surfaces the same condition as the 409). Clients MUST handle unknown values gracefully — extension specs may add new reason codes additively. See [Decision reason codes](/protocol/error-codes-and-error-handling-in-cycles#decision-reason-codes) for full semantics.

The same field is used by `/v1/decide` responses. Note that on dry_run reserve, the `BUDGET_NOT_FOUND` reason code corresponds to a condition that non-dry reserve would surface as `HTTP 404` with `error=NOT_FOUND` — the wire shape is different, but the underlying "no budget at any derived scope" condition is the same.

### balances are recommended but optional

The server may include balance snapshots in the response. These reflect the current state without any mutation — they show what balances look like without the reservation being applied.

Balance snapshots are recommended for operator visibility but are not required.

## Dry run vs decide

Both dry run and the decide endpoint (`POST /v1/decide`) evaluate budget without modifying state. But they differ in important ways:

### Scope of evaluation

- **decide** is a lightweight preflight check that returns a decision, optional caps, and affected scopes
- **dry_run** evaluates the full reservation creation path, including all normative rules that apply to real reservations

### Response completeness

- **decide** returns decision, caps, reason_code, retry_after_ms, and affected_scopes
- **dry_run** returns everything a real reservation would (except reservation_id and expires_at_ms), including balance snapshots

### Budget denial semantics

- **decide** returns a 200 response with a decision value for budget-state conditions — insufficient budget, debt, and over-limit all surface as `decision: DENY`, never as a 409. Request-validity errors are the exception to this pattern: a wrong unit still returns `400 UNIT_MISMATCH`, and malformed or unauthorized requests return their usual 4xx errors
- **dry_run** may return DENY as the decision value for insufficient budget (unlike a live reservation, which would return `409 BUDGET_EXCEEDED`); as with decide, request-validity errors still return 4xx

This is a subtle but important distinction: a live reservation with insufficient budget fails with a 409 error. A dry run with insufficient budget succeeds with a 200 response containing `decision: DENY`.

### When to use each

Use **decide** for:

- quick feasibility checks
- UI gating
- planning and routing between alternatives

Use **dry_run** for:

- full shadow-mode evaluation of reservation logic
- validating scope derivation and affected scopes
- testing budget policy before enabling enforcement
- monitoring what would happen if enforcement were live

## How to use dry run for shadow mode rollout

The typical shadow mode rollout pattern:

### Phase 1: Observe

Enable dry run on all reservation requests. Log the decisions but do not act on them.

This shows what enforcement would look like without any production impact.

### Phase 2: Alert

Configure alerts for dry run DENY decisions. Investigate whether these denials are expected or would indicate misconfigured budgets.

### Phase 3: Enforce selectively

Switch specific action classes from dry run to live enforcement. Keep others in dry run mode.

### Phase 4: Full enforcement

Once confidence is high, switch all action classes to live enforcement.

## Dry run in client code

The decorator/annotation supports dry run. When enabled, the client evaluates the reservation without holding budget. The decorated function does not execute — a `DryRunResult` is returned instead (Python) or the decision is logged for monitoring (Java). This allows teams to observe what would have happened under enforcement without affecting runtime behavior.

::: code-group
```python [Python]
from runcycles import cycles

@cycles(estimate=1000, dry_run=True)
def summarize(text: str) -> str:
    return call_llm(text)
```
```java [Java (Spring Boot)]
@Cycles(value = "1000", dryRun = true)
public String summarize(String text) {
    return chatModel.call(text);
}
```
:::

## Idempotency on dry run

Dry run requests support idempotency keys. On replay with the same key, the server returns the original response.

However, since dry run does not create a reservation, there is no reservation_id to replay. The replayed response reflects budget state at the time of the original call, not the current state.

## Practical example

A team is rolling out budget enforcement for their support bot. They configure dry run on all model calls:

```json
{
  "idempotency_key": "shadow-run-001",
  "subject": { "tenant": "acme", "app": "support-bot" },
  "action": { "kind": "llm.completion", "name": "openai:gpt-4o" },
  "estimate": { "unit": "USD_MICROCENTS", "amount": 500000 },
  "dry_run": true
}
```

The response comes back:

```json
{
  "decision": "ALLOW_WITH_CAPS",
  "affected_scopes": ["tenant:acme", "tenant:acme/app:support-bot"],
  "caps": {
    "max_tokens": 2048
  }
}
```

This tells the team: if enforcement were live, the request would be allowed but with a token cap. They can use this data to tune budgets before enabling real enforcement.

## Summary

Dry run provides full reservation-path evaluation without budget mutation:

- **decision** is returned as if the reservation were live
- **affected_scopes** is always populated, even on DENY
- **reservation_id** and **expires_at_ms** are absent
- **caps** follow the same rules as live reservations
- **balances** reflect non-mutating evaluation

Use dry run for shadow mode rollouts, policy testing, and building confidence in budget configuration before enabling enforcement.

For quick feasibility checks without full reservation evaluation, use the decide endpoint instead.

## Next steps

To explore the Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)



# Error Codes and Error Handling in Cycles

Cycles uses structured error responses with specific error codes for every failure condition.

Understanding these codes is essential for building a production integration. Each code tells the client exactly what happened and what to do about it.

## Error response format

Every error response follows the same structure:

```json
{
  "error": "BUDGET_EXCEEDED",
  "message": "Insufficient budget in scope tenant:acme",
  "request_id": "req-abc-123",
  "trace_id": "0af7651916cd43dd8448eb211c80319c",
  "details": {}
}
```

- **error** — a machine-readable error code from the fixed enum
- **message** — a human-readable explanation
- **request_id** — a unique identifier for one HTTP request
- **trace_id** — OPTIONAL. 32-hex W3C Trace Context identifier for the logical operation this request belongs to. Conformant v0.1.25.14+ runtime servers and v0.1.25.31+ admin servers populate it on every error response. See [Correlation and Tracing](/protocol/correlation-and-tracing-in-cycles).
- **details** — optional additional context

Every response — error or success — also carries an `X-Cycles-Trace-Id` HTTP response header with the same 32-hex identifier. Log both `request_id` and `trace_id` when handling errors; `trace_id` is the cross-plane join key for admin audit, events, and webhook delivery queries.

## The error codes

The runtime protocol defines 17 wire error codes (the `ErrorCode` enum in the runtime OpenAPI spec — `LIMIT_EXCEEDED` was added in spec v0.1.25.12, `TENANT_CLOSED` in spec v0.1.25.13). `TENANT_CLOSED` mirrors the governance/admin plane's lifecycle error of the same name — raised by the [tenant-close cascade](/protocol/tenant-close-cascade-semantics) against objects owned by closed tenants — so all 17 codes covered here are now part of the runtime wire contract. The admin-plane error enum has additional codes that aren't; see the admin OpenAPI spec for the full set.

Each code has a specific HTTP status code and meaning.

### INVALID_REQUEST (400)

The request is malformed or missing required fields.

Common causes:

- missing required fields (subject, action, estimate, idempotency_key)
- Subject with only `dimensions` and no standard field (tenant, workspace, app, workflow, agent, toolset)
- field values exceeding length limits
- invalid parameter values

**What to do:** fix the request. This is not retryable without changes.

### UNAUTHORIZED (401)

The `X-Cycles-API-Key` header is missing or the API key is invalid.

**What to do:** check the API key configuration. Not retryable without a valid key.

### FORBIDDEN (403)

The request is authenticated but not authorized for the target resource.

Common causes:

- Subject.tenant does not match the effective tenant derived from the API key
- attempting to commit/release/extend a reservation owned by a different tenant
- querying balances for a different tenant

**What to do:** ensure the tenant in the Subject matches the API key's tenant. Not retryable without fixing the tenant mismatch.

### NOT_FOUND (404)

The runtime plane uses a single `NOT_FOUND` wire code for all resource-not-found conditions. The `message` field carries the specific reason. Two distinct conditions surface here:

**Missing reservation.** The specified reservation ID does not exist. This is different from `RESERVATION_EXPIRED` — a 404 means the reservation was never created, while `RESERVATION_EXPIRED` means it existed but its TTL has passed. **What to do:** verify the reservation ID. If the client lost the ID, use `GET /v1/reservations` with the `idempotency_key` filter to recover it.

**Missing budget.** Returned on `POST /v1/reservations` and `POST /v1/events` when no budget ledger exists at any derived scope in any unit. The wire response looks like:

```json
{
  "error": "NOT_FOUND",
  "message": "Budget not found for provided scope: tenant:acme/workspace:prod",
  "request_id": "req-abc-123"
}
```

Distinct from `UNIT_MISMATCH (400)` — "missing budget" means *no budget exists at all*, while `UNIT_MISMATCH` means a budget exists at the scope but in a different unit than the request. **What to do:** create a budget via `POST /v1/admin/budgets` for at least one scope in the hierarchy. See [Budget Allocation and Management](/how-to/budget-allocation-and-management-in-cycles#how-budget-lookup-works-during-reservations).

On `POST /v1/decide` and `POST /v1/reservations` with `dry_run=true`, the "missing budget" condition does NOT surface as a 404. Those endpoints return `200` with `decision=DENY` and `reason_code=BUDGET_NOT_FOUND` instead — see [Decision reason codes](#decision-reason-codes) below.

### BUDGET_EXCEEDED (409)

Budget is insufficient for the requested operation.

This appears in three contexts:

1. **Reservation:** the scope does not have enough remaining budget for the estimate
2. **Commit with REJECT policy:** actual exceeds reserved
3. **Event with REJECT policy:** insufficient budget for the event amount

Note: commits with ALLOW_IF_AVAILABLE never return 409. Instead, the charge is capped to the available remaining budget.

**What to do:** depends on context:

- for reservations: degrade (smaller model, fewer tools), defer, or deny the action
- for commits: the work already happened — consider switching to ALLOW_IF_AVAILABLE or ALLOW_WITH_OVERDRAFT
- for events: adjust the amount or change the overage policy

### BUDGET_FROZEN (409)

The budget scope has been frozen by an operator. Operations that would modify the budget (reserve, commit, event) are rejected while the scope is frozen.

**What to do:** wait for the operator to unfreeze the budget, or escalate. Not retryable until the freeze is lifted.

### BUDGET_CLOSED (409)

The budget scope has been permanently closed. No further budget operations are allowed against this scope.

**What to do:** create a new budget scope or contact the operator. Not retryable against this scope.

<a id="tenant-closed-409"></a>
### TENANT_CLOSED (409)

The owning tenant has been permanently closed. Every mutating admin-plane operation on any object owned by a closed tenant — budgets, reservations, API keys, webhook subscriptions, policies — is rejected with this code. GET endpoints remain available for post-mortem audit reads.

This error is issued by the **Rule 2 — Terminal-Owner Mutation Guard** half of the cascade contract (governance-admin spec v0.1.25.29, shipped in `cycles-server-admin` v0.1.25.35; full coverage v0.1.25.36). Rule 2's counterpart — **Rule 1 — Close Cascade** — runs at tenant-close time and automatically drives owned objects to terminal states (`BudgetLedger → CLOSED`, `ApiKey → REVOKED`, open reservations → `RELEASED`, `WebhookSubscription → DISABLED`), so by the time you see this error the owned objects are already terminal. There is no way to "undo" a close; this is not a race condition that will resolve on retry.

**On the runtime plane** (runtime spec v0.1.25.13, shipped in `cycles-server` 0.1.25.47): the persisting reservation mutations — create (`dry_run` absent or `false`), commit, release, extend — return `409 TENANT_CLOSED` once the owning tenant's `CLOSED` flip is durable. For non-replay mutations it takes precedence over the reservation-state errors (`RESERVATION_FINALIZED`, `RESERVATION_EXPIRED`); same-key replays of mutations that succeeded before the close return their original stored response. Fresh (non-replay) `dry_run=true` create and `POST /v1/decide` evaluations never 409 for this condition — they return `200` with `decision=DENY` and `reason_code=TENANT_CLOSED` (see [Decision reason codes](#decision-reason-codes)). A present-but-malformed tenant record (undecodable JSON, missing or unrecognized `status`) fails closed with `500 INTERNAL_ERROR` before any mutation; deployments with no tenant records at all are not guarded. A mutation-surface `409 TENANT_CLOSED` on the evidence endpoints (persisting create, commit, release) emits an `error` CyclesEvidence envelope and stamps `cycles_evidence` on the response. Note that the cascade also revokes the tenant's API keys and the runtime auth filter rejects CLOSED-tenant keys per request, so tenant-key calls usually fail with `401` first — the 409 surfaces mainly on admin-on-behalf-of **release** (the only guarded mutation the runtime plane exposes to `X-Admin-API-Key`; the admin dual-auth allowlist is reservation list/get/release, so create/commit/extend accept tenant keys only) and, for any of the four, in the post-flip/pre-revocation race window. `cycles-server` 0.1.25.46 and earlier surface closed tenants on the runtime plane only as `401`s (revoked/rejected keys) or budget-state errors such as `BUDGET_CLOSED`.

**What to do:** the tenant and its owned objects are read-only. Create a new tenant or escalate. **Not retryable against any object owned by this tenant** — unlike `BUDGET_FROZEN` (which an operator may unfreeze), `TENANT_CLOSED` is terminal. Implement no retry logic for this error.

**For client-app developers.** If your users encounter `TENANT_CLOSED`, escalate to your platform operator — they control tenant lifecycle; a client cannot un-close a tenant. New workloads require a fresh active tenant. To proactively detect a closed tenant and surface a friendlier message before mutation attempts, call `GET /v1/admin/tenants/{tenant_id}` with admin credentials. Runtime balance and reservation reads still work for closed tenants, but their resource status fields do not report the tenant lifecycle status.

**In bulk-action responses:** rows targeting a closed tenant go into the `failed[]` bucket with `error_code=TENANT_CLOSED`; the rest of the batch continues.

See [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics) for the full Rule 1 / Rule 2 contract, affected endpoints, and Mode A vs Mode B cascade behavior.

### RESERVATION_EXPIRED (410)

The reservation's TTL plus grace period has elapsed.

The reservation has been finalized as EXPIRED and its budget has been returned to the pool.

**What to do:** create a new reservation if the work still needs to proceed. If the work already completed, the usage may need to be recorded as an event instead.

### RESERVATION_FINALIZED (409)

An operation was attempted on a reservation that is already in a terminal state (COMMITTED or RELEASED).

This typically happens when trying to extend a reservation that has already been committed.

**What to do:** no action needed on the reservation. If the extend was meant to keep a different reservation alive, check the reservation ID.

### IDEMPOTENCY_MISMATCH (409)

The same idempotency key was used with a different request payload.

This means the client sent a request with an idempotency key that was already used for a different operation.

**What to do:** use a unique idempotency key for each distinct operation. If this is a legitimate retry, ensure the request payload matches the original exactly.

### UNIT_MISMATCH (400)

The unit in the request does not match any budget stored for the derived scopes, but at least one of those scopes has a budget in a different unit.

Returned on four operations:

1. **Reserve** — `estimate.unit` does not match any budget at the derived scopes (a budget exists in a different unit)
2. **Commit** — `actual.unit` differs from the reservation's `estimate.unit`
3. **Event** — `actual.unit` does not match the budget stored for the target scope
4. **Decide** — `estimate.unit` does not match any budget at the derived scopes. This is an exception to `/decide`'s general "return `decision=DENY` (200) without 4xx" pattern, which applies only to budget-state conditions (debt, overdraft, insufficient remaining), not request-validity errors like a wrong unit.

When the cause is a wrong unit (rather than the absence of any budget at the scope), the server populates the error response's `details` object with:

- `scope` — the canonical scope identifier where the mismatch was detected
- `requested_unit` — the unit supplied by the client
- `expected_units` — array of units for which a budget does exist at that scope

so clients can self-correct without a separate lookup. `NOT_FOUND (404)` (with a `"Budget not found for provided scope: ..."` message) is reserved for the case where the target scope has no budget in **any** unit.

**What to do:** switch the request to one of the units listed in `details.expected_units`, or create a budget in the requested unit via `POST /v1/admin/budgets`.

### OVERDRAFT_LIMIT_EXCEEDED (409)

Appears in two contexts:

1. **During commit:** when `overage_policy=ALLOW_WITH_OVERDRAFT` and `(current_debt + delta) > overdraft_limit`
2. **During reservation:** when the scope is in over-limit state (`is_over_limit=true`) due to prior concurrent commits pushing debt past the limit

**What to do:**

- if during commit: the debt limit has been reached. The work already happened. An operator needs to fund the scope.
- if during reservation: the scope is blocked. Wait for debt to be repaid, or escalate to an operator. The client should retry with exponential backoff.

### DEBT_OUTSTANDING (409)

A new reservation was attempted against a scope that has outstanding debt (debt > 0) and no overdraft limit configured (overdraft_limit is absent or 0).

When an `overdraft_limit > 0` is configured, debt within the limit does not block new reservations. Only scopes without an overdraft limit treat any debt as blocking.

**What to do:** wait for debt to be repaid through budget funding, or configure an overdraft limit if debt within a limit is acceptable. Retry with exponential backoff, or escalate to an operator.

Note: when `is_over_limit=true`, the server returns `OVERDRAFT_LIMIT_EXCEEDED` instead of `DEBT_OUTSTANDING`, even if debt > 0. `OVERDRAFT_LIMIT_EXCEEDED` takes precedence.

### MAX_EXTENSIONS_EXCEEDED (409)

The tenant's `max_reservation_extensions` limit has been reached for this reservation. No further extensions are allowed.

**What to do:** commit or release the reservation. If more time is needed, create a new reservation after committing the current one.

### LIMIT_EXCEEDED (429)

The client has exceeded the server's rate limit. Added in spec v0.1.25.12, mirroring the governance-plane code of the same name; the reference runtime server enforces it on the **public (unauthenticated)** endpoints — `GET /v1/evidence/*` and the CyclesEvidence JWKS — since v0.1.25.46 (default 300 requests/minute per client IP, per instance; see [Public endpoint rate limiting](/configuration/server-configuration-reference-for-cycles#public-endpoint-rate-limiting-v0-1-25-46)). Authenticated `/v1` endpoints are not rate limited by the reference server (abuse there is key-attributable).

The 429 response carries throttling headers alongside the standard correlation headers:

- `Retry-After` — seconds to wait before retrying
- `X-RateLimit-Reset` — when the current window resets
- `X-RateLimit-Remaining: 0`

**What to do:** wait for the `Retry-After` interval, then retry. If you hit this limit during legitimate operation, raise `CYCLES_PUBLIC_RATE_LIMIT_REQUESTS_PER_MINUTE` or rate-limit at your ingress instead.

### INTERNAL_ERROR (500)

An unexpected server error occurred.

Since `cycles-server` 0.1.25.47 this is also the deliberate fail-closed response when a tenant record exists but its status cannot be determined (undecodable JSON, non-object, missing or unrecognized `status`) — the closed-tenant guard refuses to treat a corrupt governance record as an open tenant, on the dry-run/decide surface too. That variant will not resolve on retry; the operator must repair the tenant record.

**What to do:** retry unexpected or transient internal errors with exponential backoff. If the error persists, contact the Cycles server operator. Do not keep retrying the malformed-tenant-record variant described above; it requires operator repair.

## Error handling by operation

### Reserve errors

| Error | HTTP | Meaning |
|---|---|---|
| BUDGET_EXCEEDED | 409 | Insufficient budget |
| BUDGET_FROZEN | 409 | Budget scope is frozen |
| BUDGET_CLOSED | 409 | Budget scope is permanently closed |
| TENANT_CLOSED | 409 | Owning tenant is closed (cycles-server 0.1.25.47+, spec v0.1.25.13). Persisting create only — a fresh `dry_run=true` returns `200 decision=DENY reason_code=TENANT_CLOSED` instead |
| OVERDRAFT_LIMIT_EXCEEDED | 409 | Scope is over-limit |
| DEBT_OUTSTANDING | 409 | Scope has unresolved debt (no overdraft limit configured) |
| IDEMPOTENCY_MISMATCH | 409 | Same key, different payload |
| NOT_FOUND | 404 | No budget exists at any derived scope in any unit (message: `"Budget not found for provided scope: ..."`) |
| UNIT_MISMATCH | 400 | `estimate.unit` does not match any budget at the derived scopes (budget exists in a different unit) |
| INVALID_REQUEST | 400 | Malformed request |
| UNAUTHORIZED | 401 | Invalid API key |
| FORBIDDEN | 403 | Tenant mismatch |

### Decide errors

| Error | HTTP | Meaning |
|---|---|---|
| UNIT_MISMATCH | 400 | `estimate.unit` does not match any budget at the derived scopes (budget exists in a different unit) |
| INVALID_REQUEST | 400 | Malformed request |
| UNAUTHORIZED | 401 | Invalid API key |
| FORBIDDEN | 403 | Tenant mismatch |
| IDEMPOTENCY_MISMATCH | 409 | Same key, different payload |

Note: decide returns `200` with `decision: DENY` for budget-state conditions (insufficient remaining, debt, overdraft, and the "no budget exists at any scope" case — surfaced via `reason_code` from the [DecisionReasonCode enum](#decision-reason-codes)), not a `409` or `404` error. The same holds for a closed owning tenant (cycles-server 0.1.25.47+, spec v0.1.25.13): a fresh (non-replay) evaluation returns `200 decision=DENY reason_code=TENANT_CLOSED`, never `409 TENANT_CLOSED` — though a present-but-malformed tenant record fails closed with `500 INTERNAL_ERROR`. Request-validity errors like `UNIT_MISMATCH` are still returned as 400. The same applies to `POST /v1/reservations` when `dry_run=true`.

### Commit errors

| Error | HTTP | Meaning |
|---|---|---|
| BUDGET_EXCEEDED | 409 | Actual exceeds budget (REJECT only) |
| BUDGET_FROZEN | 409 | Budget scope is frozen |
| BUDGET_CLOSED | 409 | Budget scope is permanently closed |
| TENANT_CLOSED | 409 | Owning tenant is closed (cycles-server 0.1.25.47+, spec v0.1.25.13); takes precedence over reservation-state errors for non-replay requests |
| OVERDRAFT_LIMIT_EXCEEDED | 409 | Debt would exceed limit (ALLOW_WITH_OVERDRAFT) |
| RESERVATION_EXPIRED | 410 | Past TTL + grace period |
| RESERVATION_FINALIZED | 409 | Already committed or released |
| UNIT_MISMATCH | 400 | Unit differs from reservation |
| NOT_FOUND | 404 | Reservation never existed |
| IDEMPOTENCY_MISMATCH | 409 | Same key, different payload |
| UNAUTHORIZED | 401 | Invalid API key |
| FORBIDDEN | 403 | Reservation owned by different tenant |

### Release errors

| Error | HTTP | Meaning |
|---|---|---|
| TENANT_CLOSED | 409 | Owning tenant is closed (cycles-server 0.1.25.47+, spec v0.1.25.13); takes precedence over reservation-state errors for non-replay requests |
| RESERVATION_EXPIRED | 410 | Past TTL + grace period |
| RESERVATION_FINALIZED | 409 | Already committed or released |
| IDEMPOTENCY_MISMATCH | 409 | Same key, different payload |
| NOT_FOUND | 404 | Reservation never existed |
| UNAUTHORIZED | 401 | Invalid API key |
| FORBIDDEN | 403 | Reservation owned by different tenant |

### Extend errors

| Error | HTTP | Meaning |
|---|---|---|
| INVALID_REQUEST | 400 | Missing or invalid fields |
| TENANT_CLOSED | 409 | Owning tenant is closed (cycles-server 0.1.25.47+, spec v0.1.25.13); takes precedence over reservation-state errors for non-replay requests |
| RESERVATION_EXPIRED | 410 | Past TTL (no grace period for extend) |
| RESERVATION_FINALIZED | 409 | Already committed or released |
| MAX_EXTENSIONS_EXCEEDED | 409 | Tenant max_reservation_extensions limit reached |
| IDEMPOTENCY_MISMATCH | 409 | Same key, different payload |
| NOT_FOUND | 404 | Reservation never existed |
| UNAUTHORIZED | 401 | Invalid API key |
| FORBIDDEN | 403 | Reservation owned by different tenant |

### Event errors

| Error | HTTP | Meaning |
|---|---|---|
| BUDGET_EXCEEDED | 409 | Insufficient budget (REJECT only) |
| BUDGET_FROZEN | 409 | Budget scope is frozen |
| BUDGET_CLOSED | 409 | Budget scope is permanently closed |
| TENANT_CLOSED | 409 | Owning tenant is closed (cycles-server 0.1.25.47+, spec v0.1.25.13); applies to fresh event requests because event creation is a persisting budget debit |
| OVERDRAFT_LIMIT_EXCEEDED | 409 | Debt would exceed limit (ALLOW_WITH_OVERDRAFT) |
| NOT_FOUND | 404 | No budget exists at any derived scope in any unit (message: `"Budget not found for provided scope: ..."`) |
| UNIT_MISMATCH | 400 | `actual.unit` does not match any budget at the target scope (budget exists in a different unit) |
| INVALID_REQUEST | 400 | Malformed request |
| UNAUTHORIZED | 401 | Invalid API key |
| FORBIDDEN | 403 | Tenant mismatch |
| IDEMPOTENCY_MISMATCH | 409 | Same key, different payload |

For a fresh `/v1/events` request observed after the tenant's `CLOSED` flip, the runtime returns `409 TENANT_CLOSED`. Event creation has no dry-run mode, so this condition never becomes a `200 decision=DENY` response. A same-key replay of an event that succeeded before the close returns its original stored `201` response, and a malformed or undeterminable tenant record fails closed with `500 INTERNAL_ERROR`. In normal operation the auth filter may return `401` first because the close cascade also revokes tenant API keys.

## Decision reason codes

Separately from the 4xx error code list, `POST /v1/decide` and `POST /v1/reservations` with `dry_run=true` may return `200 OK` with `decision: DENY` and a machine-readable `reason_code`. As of v0.1.25, `DecisionReasonCode` is an **open string** (was a closed enum in v0.1.24 and earlier — widened so future extension specs can add reason codes without a breaking protocol bump). Documented known values:

| reason_code | Meaning |
|---|---|
| `BUDGET_EXCEEDED` | Remaining amount insufficient on at least one derived scope (evaluated against the requested `estimate.amount`). |
| `BUDGET_FROZEN` | A derived scope has a budget in `FROZEN` status (operator-set, no mutations allowed). |
| `BUDGET_CLOSED` | A derived scope has a budget in `CLOSED` status (permanently closed). |
| `BUDGET_NOT_FOUND` | No budget exists at any derived scope in the requested unit. On non-dry reserve and `/v1/events` paths this same underlying condition surfaces as `HTTP 404` with `error=NOT_FOUND` instead. |
| `OVERDRAFT_LIMIT_EXCEEDED` | Either `debt + delta > overdraft_limit` on commit, OR the scope is in over-limit state (`is_over_limit=true`) and no new reservations are permitted until reconciled. |
| `DEBT_OUTSTANDING` | A derived scope has `debt > 0` and `overdraft_limit == 0` (no policy permits further debt accrual). |
| `TENANT_CLOSED` | The owning tenant's status is `CLOSED` (deployments with a governance plane; added in spec v0.1.25.13, emitted by cycles-server 0.1.25.47+ on fresh dry-run/decide evaluations). The persisting mutation surface reports the same condition as `HTTP 409` with `error=TENANT_CLOSED` instead — see [TENANT_CLOSED (409)](#tenant-closed-409). |

**Why this is a separate enum.** The 4xx error codes surface request-level failures in the `error` field. Decision reason codes surface budget-state outcomes in the `reason_code` field on successful HTTP responses. Some labels overlap (e.g. `BUDGET_EXCEEDED`) because the same underlying condition is reported differently depending on the endpoint: `/decide` and dry-run reserve surface it as a non-4xx DENY decision, while non-dry reserve surfaces it as a `409` error.

**Forward compatibility.** Because `DecisionReasonCode` is an open string (since v0.1.25), **clients MUST handle unknown values gracefully** — treat as DENY, log the raw string, do not crash on enum parsing. Known values above are stable; future values will always be additive (e.g., v0.1.26 extension specs may emit `ACTION_QUOTA_EXCEEDED`, `ACTION_KIND_DENIED`, `ACTION_KIND_NOT_ALLOWED`).

## Idempotency and error handling

Errors interact with idempotency in specific ways:

- **Successful replay:** if you retry a request with the same idempotency key and payload, you get the original successful response. The operation is not applied again.
- **Payload mismatch:** if you reuse a key with a different payload, you get `409 IDEMPOTENCY_MISMATCH`.
- **Failed original:** if the original request failed (e.g., BUDGET_EXCEEDED), retrying with the same key sends a fresh request. Idempotency only applies to successful operations.

## Correlation identifiers

Every error response carries two server-generated identifiers:

- **`request_id`** — unique per HTTP request. Useful for correlating errors with server logs, debugging with the Cycles operator, and tracking specific failures in client-side monitoring.
- **`trace_id`** — 32-hex W3C Trace Context identifier. OPTIONAL on the schema; populated by conformant v0.1.25.14+ runtime and v0.1.25.31+ admin servers. Scopes a logical operation that may span multiple HTTP requests. Also echoed in the `X-Cycles-Trace-Id` response header.

Log both when handling errors. `trace_id` is usually the right choice for cross-plane root-cause analysis:

```bash
# Find everything that happened under one trace
GET /v1/admin/audit/logs?trace_id=<32-hex>
GET /v1/admin/events?trace_id=<32-hex>
```

`request_id` narrows to the side effects of one specific HTTP call. See [Correlation and Tracing](/protocol/correlation-and-tracing-in-cycles) for the full contract.

## Summary

Cycles provides 17 specific error codes that tell the client exactly what went wrong:

- **400** for request validation issues (INVALID_REQUEST, UNIT_MISMATCH)
- **401** for authentication failures (UNAUTHORIZED)
- **403** for authorization failures (FORBIDDEN)
- **404** for missing resources (NOT_FOUND) — covers both missing reservations and missing budgets, distinguished by the `message` field
- **409** for budget and state conflicts (BUDGET_EXCEEDED, BUDGET_FROZEN, BUDGET_CLOSED, TENANT_CLOSED, OVERDRAFT_LIMIT_EXCEEDED, DEBT_OUTSTANDING, RESERVATION_FINALIZED, IDEMPOTENCY_MISMATCH, MAX_EXTENSIONS_EXCEEDED)
- **410** for expired reservations (RESERVATION_EXPIRED)
- **429** for rate limiting on public endpoints (LIMIT_EXCEEDED)
- **500** for server errors (INTERNAL_ERROR)

Additionally, `/v1/decide` and dry-run reserve surface budget-state conditions via a `reason_code` field on `200 DENY` responses rather than as 4xx errors. These values come from a separate [DecisionReasonCode](#decision-reason-codes) enum — distinct from the 4xx error code list.

Handling these codes correctly is the difference between a fragile integration and a production-grade one.

## Debugging with `trace_id`

When an error response carries `trace_id`, the fastest way to reconstruct the full operation is a three-call walk across the admin plane:

```bash
TID=<value from X-Cycles-Trace-Id header or error-body trace_id>
curl -s "http://localhost:7979/v1/admin/audit/logs?trace_id=$TID" -H "X-Admin-API-Key: $ADMIN_KEY"
curl -s "http://localhost:7979/v1/admin/events?trace_id=$TID"     -H "X-Admin-API-Key: $ADMIN_KEY"
```

See [Correlation and Tracing](/protocol/correlation-and-tracing-in-cycles) for the full contract (W3C Trace Context precedence, outbound headers on webhook deliveries, cross-plane propagation rules).

## Next steps

To explore the Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- [Correlation and Tracing](/protocol/correlation-and-tracing-in-cycles) — `trace_id` as the cross-plane join key for debugging
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)



# Event Payloads Reference

This page documents the payload structure for every webhook event Cycles can emit. Each event wraps a standard envelope with an event-specific `data` object.

::: info Currently Emitted Events
The v0.1.25 Admin API `EventType` enum registers **51 event types** total across seven categories (budget: 17, reservation: 6, tenant: 6, api_key: 7, policy: 3, webhook: 7, system: 5). Events marked as **Planned** below have their type registered in the protocol but are not yet emitted by any service.

**Registered enum values currently emitted** (count toward the 51 total):

- **Reservation:** `reservation.denied`, `reservation.expired`, `reservation.commit_overage` (runtime).
- **Budget:** runtime emits `budget.exhausted`, `budget.over_limit_entered`, and `budget.debt_incurred`; admin emits `budget.created`, `budget.updated`, `budget.frozen`, `budget.unfrozen`, `budget.funded`, `budget.debited`, `budget.reset`, `budget.reset_spent`, and `budget.debt_repaid`; tenant close emits `budget.closed_via_tenant_cascade`.
- **Tenant:** `tenant.created`, `tenant.updated` (current admin implementation); `tenant.suspended`, `tenant.reactivated`, `tenant.closed` (admin v0.1.25.38+, single-op + bulk-action paths).
- **API key:** `api_key.created`, `api_key.revoked`, and `api_key.auth_failed` (admin v0.1.25+); `api_key.permissions_changed` (admin v0.1.25.7+).
- **Policy:** `policy.created` and `policy.updated` (admin v0.1.25+).
- **Webhook:** `webhook.created`, `webhook.updated`, `webhook.paused`, `webhook.resumed`, `webhook.deleted` (admin v0.1.25.39+); `webhook.disabled` (events service auto-disable v0.1.25.11+). All six webhook lifecycle types were added in spec v0.1.25.33 — see the [Webhook Lifecycle Events](#webhook-lifecycle-events) section below.
- **System:** `system.webhook_test` is sent directly by the admin webhook-test endpoint; `system.webhook_delivery_failed` is persisted by the events service after retry exhaustion (events v0.1.25.21+).
- **Tenant-close cascade fan-out:** `budget.closed_via_tenant_cascade`, `reservation.released_via_tenant_cascade`, `api_key.revoked_via_tenant_cascade`, `webhook.disabled_via_tenant_cascade` (admin v0.1.25.35+; declared in the governance spec's enum since revision v0.1.25.35) — see [Tenant-Close Cascade Events](#tenant-close-cascade-events-governance-spec-v0-1-25-35) below.

Registered values not named above remain planned unless a later section says otherwise.

**Historical additive runtime payloads** (documented in the v0.1.25.3 release history, but not present in the current runtime `EventType` model or emitted by current controller paths):

- Reservation lifecycle samples: `reservation.reserved`, `reservation.committed`, `reservation.released`, `reservation.extended`.
- Runtime ledger application: `event.applied`.

See the [Event Emission Summary](#event-emission-summary) at the bottom for the full per-category breakdown.
:::

## Standard Envelope

Every event shares this envelope structure. The `data` field varies by event type.

```json
{
  "event_id": "evt_a1b2c3d4e5f67890",
  "event_type": "budget.exhausted",
  "category": "budget",
  "timestamp": "2026-04-01T14:32:00.123Z",
  "tenant_id": "acme-corp",
  "scope": "tenant:acme-corp/workspace:prod",
  "source": "cycles-server",
  "actor": {
    "type": "api_key",
    "key_id": "key_abc123",
    "source_ip": "10.0.1.50"
  },
  "data": { },
  "correlation_id": "3f2a9c14e0b7d5a1",
  "request_id": "req_789",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "metadata": {}
}
```

### Envelope fields

| Field | Type | Always present | Description |
|---|---|---|---|
| `event_id` | string | Yes | Unique event identifier (format: `evt_*`). Use for deduplication. |
| `event_type` | string | Yes | Dotted event name (e.g., `budget.exhausted`) |
| `category` | string | Yes | One of: `budget`, `reservation`, `tenant`, `api_key`, `policy`, `webhook`, `system` (the `webhook` value was added in spec v0.1.25.34) |
| `timestamp` | string | Yes | ISO 8601 UTC timestamp |
| `tenant_id` | string | Yes | Tenant ID (system events use `__system__`) |
| `scope` | string | When applicable | Full scope path (e.g., `tenant:acme-corp/workspace:prod`) |
| `source` | string | Yes | Emitting service: `cycles-server` (runtime events), `cycles-admin` (admin-plane events including bulk-action emits and webhook lifecycle events since v0.1.25.38/.39), or `cycles-events` (dispatcher-emitted `webhook.disabled` on auto-disable, v0.1.25.11). |
| `actor` | object | When applicable | Who triggered: `type` (`api_key`, `admin`, `system`, `scheduler`), `key_id`, `source_ip` |
| `data` | object | Varies | Event-specific payload (see below). Some events emit `null`. |
| `correlation_id` | string | When applicable | Server-managed family key. The YAML requires deterministic runtime event clusters, but the current reference runtime leaves this absent on implemented runtime emits. Selected admin operations populate explicit IDs such as `webhook_create:<id>`, bulk-action IDs, and cascade IDs. |
| `request_id` | string | When provided | From `X-Request-Id` header on originating request |
| `trace_id` | string | When provided | W3C Trace Context-compatible correlation identifier (32 lowercase hex characters). Links the event to the originating request, its audit entry, and sibling events within the same logical operation. |
| `metadata` | object | When provided | Operator-defined key-value pairs |

---

## Reservation Events

### `reservation.reserved` — Historical Additive Payload (v0.1.25.3)

**Historical trigger:** A reservation was created successfully.

The current reference runtime does not emit this event. Query reservation state and keep application telemetry for successful reserve operations.

---

### `reservation.committed` — Historical Additive Payload (v0.1.25.3)

**Historical trigger:** A reservation was committed with actual spend recorded.

The current reference runtime does not emit this event. A commit that requests more than the estimate can emit `reservation.commit_overage`, and budget-state changes can emit the implemented budget events.

---

### `reservation.released` — Historical Additive Payload (v0.1.25.3)

**Historical trigger:** A reservation was cancelled.

The current reference runtime does not emit this event. An admin-on-behalf-of release does write its required audit entry; ordinary release state remains queryable through the reservation API.

---

### `reservation.extended` — Historical Additive Payload (v0.1.25.3)

**Historical trigger:** A reservation TTL was extended via heartbeat.

The current reference runtime does not emit this event. Query the reservation for current expiry state and log successful extensions in the application when needed.

---

### `reservation.denied` — Currently Emitted

**Trigger:** `POST /v1/decide` or a reservation request with `dry_run: true` returns `decision: DENY`.

**Emitted from:** `POST /v1/reservations` when the nonpersisting dry-run response is DENY, and `POST /v1/decide` when its response is DENY. A live reservation denial is an HTTP error such as `409 BUDGET_EXCEEDED`; the current controller does not emit `reservation.denied` for that exception path. Monitor `cycles_reservations_reserve_total{decision="DENY"}` or application errors for live denial rate.

```json
{
  "event_type": "reservation.denied",
  "data": {
    "scope": "tenant:acme-corp/workspace:prod/workflow:support",
    "unit": "USD_MICROCENTS",
    "reason_code": "BUDGET_EXCEEDED",
    "requested_amount": 500000,
    "remaining": 100000,
    "action": {
      "kind": "llm.chat",
      "name": "support-reply"
    },
    "subject": {
      "tenant": "acme-corp",
      "workspace": "prod",
      "workflow": "support"
    }
  }
}
```

::: tip Fields populated at emission time
The governance schema defines 9 fields. The current server emitter populates `scope`, `unit`, `reason_code`, `requested_amount`, `action`, and `subject`; a denied reservation dry run also derives `remaining` from returned balances, while `/decide` currently omits `remaining`. `policy_id` and `deny_detail` remain unpopulated.
:::

| Field | Type | Populated | Description |
|---|---|---|---|
| `scope` | string | Yes | Scope path that denied the reservation |
| `reason_code` | string | Yes | Why denied. Known values: `BUDGET_EXCEEDED`, `OVERDRAFT_LIMIT_EXCEEDED`, `DEBT_OUTSTANDING`, `BUDGET_FROZEN`, `BUDGET_CLOSED`, and — from cycles-server 0.1.25.47 (spec v0.1.25.13) — `TENANT_CLOSED` on fresh dry-run/decide DENYs for a closed owning tenant. Open string — extensions (v0.1.26+) may emit additional values such as `ACTION_QUOTA_EXCEEDED`, `ACTION_KIND_DENIED`, `ACTION_KIND_NOT_ALLOWED`. |
| `requested_amount` | number | Yes | Amount the reservation requested |
| `unit` | string | Yes | Budget unit (`USD_MICROCENTS`, `TOKENS`, `CREDITS`, `RISK_POINTS`) |
| `remaining` | number | Dry-run reserve only | Minimum remaining amount derived from returned balances; omitted by the current `/decide` emitter |
| `action` | object | Yes | Action metadata from the evaluation request |
| `subject` | object | Yes | Subject metadata from the evaluation request |
| `policy_id` | string | Not yet | Policy ID that caused the denial, when applicable (added v0.1.25.8) |
| `deny_detail` | object | Not yet | Operator-grade structured context (added v0.1.25.8). Populated by extensions; may include `quota_violation`, `blocked_by_policy`, `blocked_by_scope`, `suggested_fix`, `budget_remaining`. |

---

### `reservation.commit_overage` — Currently Emitted

**Trigger:** A commit's actual cost exceeds the original reservation estimate.

**Emitted from:** `POST /v1/reservations/{id}/commit` (when `actual > estimated`)

```json
{
  "event_type": "reservation.commit_overage",
  "scope": "tenant:acme/workflow:support",
  "data": {
    "reservation_id": "res_a1b2c3d4",
    "scope": "tenant:acme/workflow:support",
    "unit": "USD_MICROCENTS",
    "estimated_amount": 400000,
    "actual_amount": 480000,
    "overage": 80000,
    "overage_policy": "ALLOW_IF_AVAILABLE",
    "debt_incurred": 0
  }
}
```

::: tip Fields populated at emission time
As of cycles-server v0.1.25.46, the emission populates **all 8** data fields and sets the envelope `scope` to the reservation's scope path — so `commit_overage` participates in scope filtering like any other scoped event. (Earlier releases populated only `reservation_id` and `actual_amount`, with a null envelope scope.)
:::

| Field | Type | Populated | Description |
|---|---|---|---|
| `reservation_id` | string | Yes | The reservation that exceeded its estimate |
| `scope` | string | Yes | Affected scope path (also set on the envelope) |
| `unit` | string | Yes | Budget unit |
| `estimated_amount` | number | Yes | Original reservation estimate |
| `actual_amount` | number | Yes | Actual cost committed |
| `overage` | number | Yes | Amount by which actual exceeded estimate |
| `overage_policy` | string | Yes | Policy applied: `REJECT`, `ALLOW_IF_AVAILABLE`, `ALLOW_WITH_OVERDRAFT` |
| `debt_incurred` | number | Yes | Debt created (0 unless `ALLOW_WITH_OVERDRAFT`) |

---

### `reservation.expired` — Currently Emitted

**Trigger:** A reservation TTL expires without being committed or released.

**Emitted from:** Background expiry sweeper (runs every 5 seconds by default)

```json
{
  "event_type": "reservation.expired",
  "data": {
    "reservation_id": "res_d4e5f678",
    "scope": "tenant:acme-corp/workspace:prod",
    "unit": "USD_MICROCENTS",
    "estimated_amount": 200000,
    "created_at": "2026-04-01T14:30:00.000Z",
    "expired_at": "2026-04-01T14:35:30.000Z",
    "ttl_ms": 300000,
    "extensions_used": 0
  }
}
```

| Field | Type | Description |
|---|---|---|
| `reservation_id` | string | The expired reservation |
| `scope` | string | Affected scope path |
| `unit` | string | Budget unit |
| `estimated_amount` | number | Amount that was held by the reservation |
| `created_at` | string | When the reservation was created (ISO 8601) |
| `expired_at` | string | When the reservation expired (ISO 8601) |
| `ttl_ms` | number | Effective TTL in milliseconds (computed as `expired_at - created_at`; includes extensions) |
| `extensions_used` | number | How many times the reservation was extended before expiry |

---

### `reservation.denial_rate_spike` — Planned

**Trigger:** Denial rate exceeds configured threshold within a rolling window.

::: warning Not Yet Emitted
This event type is defined in the protocol but not yet emitted by the Cycles server. It will be implemented in a future release.
:::

---

### `reservation.expiry_rate_spike` — Planned

**Trigger:** Expiry rate exceeds configured threshold within a rolling window.

::: warning Not Yet Emitted
This event type is defined in the protocol but not yet emitted by the Cycles server. It will be implemented in a future release.
:::

---

## Budget Events

### Historical threshold aliases

Earlier v0.1.25.x builds documented additive `budget.approaching_limit`, `budget.at_limit`, and `budget.over_limit` aliases. They are not in the current runtime `EventType` model and the current `EventEmitterService` does not emit them. Use `budget.exhausted` for the implemented zero-remaining transition and calculate earlier utilization alerts from balances or metrics. The registered `budget.threshold_crossed` type remains unimplemented.

---

### `budget.reset_spent` — Currently Emitted (v0.1.25.18)

**Trigger:** An admin operator issues a `RESET_SPENT` funding operation on `POST /v1/admin/budgets/fund`.

**Emitted from:** `cycles-server-admin`. Distinct from `budget.reset` — `RESET` resizes the allocated ceiling and preserves `spent`; `RESET_SPENT` additionally clears (or overrides) `spent` for billing-period rollover.

The payload is an `EventDataBudgetLifecycle` with `spent` and `reserved` fields on `BudgetState`, plus an optional `spent_override_provided` boolean flag on the outer payload (`true` when the operator supplied an explicit `spent` value).

See [Rolling over billing periods with RESET_SPENT](/how-to/rolling-over-billing-periods-with-reset-spent) for operator guidance.

---

### `event.applied` — Historical Additive Payload (v0.1.25.3)

**Historical trigger:** A direct debit via `POST /v1/events` was applied successfully.

The current reference runtime does not emit `event.applied`. It returns the applied direct-debit response and can emit implemented budget-state events when that debit changes a ledger.

---

### `budget.exhausted` — Currently Emitted

**Trigger:** A budget's remaining amount transitions from above zero to zero after a reservation, commit, or direct debit.

**Emitted from:** `EventEmitterService.emitBalanceEvents()` (when pre-operation remaining is above zero and post-operation remaining is zero)

```json
{
  "event_type": "budget.exhausted",
  "data": {
    "scope": "tenant:acme-corp/workspace:prod",
    "unit": "USD_MICROCENTS",
    "threshold": 1.0,
    "utilization": 1.0,
    "allocated": 10000000,
    "remaining": 0,
    "spent": 9000000,
    "reserved": 1000000,
    "direction": "rising"
  }
}
```

| Field | Type | Description |
|---|---|---|
| `scope` | string | Affected scope path |
| `unit` | string | Budget unit |
| `threshold` | number | `1.0` for the implemented exhaustion transition |
| `utilization` | number | `(spent + reserved) / allocated` when allocated is positive |
| `allocated` | number | Current allocated amount |
| `remaining` | number | `0` for this event |
| `spent` | number | Current spent amount |
| `reserved` | number | Current reserved amount |
| `direction` | string | `rising` |

The envelope also identifies the tenant, actor, and request context. Query the balance API before remediation because later operations can change the ledger after the event is emitted.

---

### `budget.over_limit_entered` — Currently Emitted

**Trigger:** Debt exceeds the configured `overdraft_limit` on a budget with `ALLOW_WITH_OVERDRAFT` policy.

**Emitted from:** `EventEmitterService.emitBalanceEvents()` (when `is_over_limit` transitions to `true`)

```json
{
  "event_type": "budget.over_limit_entered",
  "data": {
    "scope": "tenant:acme-corp/workspace:prod",
    "unit": "USD_MICROCENTS",
    "debt": 1500000,
    "overdraft_limit": 1000000,
    "is_over_limit": true,
    "debt_utilization": 1.5
  }
}
```

| Field | Type | Description |
|---|---|---|
| `scope` | string | Affected scope path |
| `unit` | string | Budget unit |
| `debt` | number | Current debt amount |
| `overdraft_limit` | number | Configured overdraft ceiling |
| `is_over_limit` | boolean | Always `true` for this event |
| `debt_utilization` | number | Ratio: `debt / overdraft_limit` |

---

### `budget.debt_incurred` — Currently Emitted

**Trigger:** A reservation commit or direct debit creates new debt via `ALLOW_WITH_OVERDRAFT`.

**Emitted from:** `EventEmitterService.emitBalanceEvents()` (when new debt is created)

```json
{
  "event_type": "budget.debt_incurred",
  "data": {
    "scope": "tenant:acme-corp/workspace:prod",
    "unit": "USD_MICROCENTS",
    "reservation_id": "res_a1b2c3d4",
    "debt_incurred": 250000,
    "total_debt": 750000,
    "overdraft_limit": 1000000,
    "overage_policy": "ALLOW_WITH_OVERDRAFT"
  }
}
```

| Field | Type | Description |
|---|---|---|
| `scope` | string | Affected scope path |
| `unit` | string | Budget unit |
| `reservation_id` | string | Reservation whose commit caused the debt; omitted for a direct debit |
| `debt_incurred` | number | New debt created on this scope by the operation |
| `total_debt` | number | Total accumulated debt on this scope |
| `overdraft_limit` | number | Configured overdraft ceiling |
| `overage_policy` | string | Policy applied (`ALLOW_WITH_OVERDRAFT`) |

---

### Planned Budget Events

The following registered budget events are not emitted by the current reference services:

| Event Type | Trigger |
|---|---|
| `budget.closed` | Budget permanently closed |
| `budget.threshold_crossed` | Utilization crossed configured threshold (e.g., 80%, 95%) |
| `budget.over_limit_exited` | Debt dropped below overdraft limit after repayment |
| `budget.burn_rate_anomaly` | Spend rate exceeds baseline multiplier within window |

---

## Tenant-Close Cascade Events (governance spec v0.1.25.35+)

Four event kinds are emitted by the reference admin server as side effects of a `* → CLOSED` tenant transition (Rule 1 — Close Cascade; see [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics) for the full contract). All four share the `_via_tenant_cascade` suffix and carry a server-composed `correlation_id` of the form `tenant_close_cascade:<tenant_id>:<request_id>`, so subscribers can correlate cascade side effects to the operator action that triggered them (audit rows for the same operation join via `request_id`/`trace_id`).

These four event names are **declared in the governance spec's `EventType` enum** since document revision v0.1.25.35 (raising the registered enum to 51 values), so cascade Events validate against `Event.event_type` and can be targeted with `event_type=` filters and webhook `event_types` lists like any other lifecycle event. Note the normative strength: the per-object cascade **audit entries** are a MUST (their `event_kind` values are RESERVED in the spec), while emitting the corresponding Event-stream records is a SHOULD — so non-reference servers may not emit them, and consumers should still ignore unrecognized event types gracefully. Tenant self-service subscriptions filter by category, so a tenant subscribed to `budget` or `reservation` events will receive the corresponding cascade events from the reference server in practice.

Shipped in `cycles-server-admin` v0.1.25.35 (initial Mode B cascade) / v0.1.25.36 (full Rule 2 guard coverage).

All four kinds share one payload shape (`EventDataTenantCascade`, governance spec v0.1.25.35): exactly one of `ledger_id` / `subscription_id` / `key_id` identifies the transitioned object (matching the event's category), alongside `prior_status` / `new_status` and `cascade_reason: "tenant_closed"`. The reservation aggregate is the exception — it identifies the drained budget via `ledger_id` and carries `released_amount` instead of a status transition.

### `budget.closed_via_tenant_cascade`

Emitted once per owned `BudgetLedger` when the tenant closes. The per-budget `BudgetLedger.status` flips to `CLOSED` and `closed_at` is stamped; the final balance snapshot is preserved for audit.

```json
{
  "event_id": "evt_...",
  "event_type": "budget.closed_via_tenant_cascade",
  "category": "budget",
  "timestamp": "2026-04-20T12:00:00Z",
  "tenant_id": "acme-corp",
  "scope": "tenant:acme-corp/workspace:prod",
  "source": "cycles-admin",
  "actor": {
    "type": "admin"
  },
  "data": {
    "ledger_id": "led_...",
    "scope": "tenant:acme-corp/workspace:prod",
    "unit": "USD_MICROCENTS",
    "prior_status": "ACTIVE",
    "new_status": "CLOSED",
    "cascade_reason": "tenant_closed"
  },
  "correlation_id": "tenant_close_cascade:acme-corp:req_...",
  "trace_id": "<same as originating>"
}
```

### `reservation.released_via_tenant_cascade`

Emitted as a **ledger-level aggregate** when the tenant closes: one event per closed budget with `reserved > 0`, carrying the aggregate `released_amount` (not one per reservation). Reason `tenant_closed`; no overage debt is recorded; the full reserved amount returns to the (now-closed) budget's balance snapshot.

```json
{
  "event_id": "evt_...",
  "event_type": "reservation.released_via_tenant_cascade",
  "category": "reservation",
  "tenant_id": "acme-corp",
  "data": {
    "ledger_id": "led_...",
    "scope": "tenant:acme-corp/workspace:prod",
    "unit": "USD_MICROCENTS",
    "released_amount": 250000,
    "cascade_reason": "tenant_closed"
  },
  "correlation_id": "tenant_close_cascade:acme-corp:req_...",
  "trace_id": "<same as originating>"
}
```

### `api_key.revoked_via_tenant_cascade`

Emitted once per owned `ApiKey` when the tenant closes. The per-key `ApiKey.status` flips to `REVOKED` and `revoked_at` is stamped.

```json
{
  "event_id": "evt_...",
  "event_type": "api_key.revoked_via_tenant_cascade",
  "category": "api_key",
  "tenant_id": "acme-corp",
  "data": {
    "key_id": "key_...",
    "prior_status": "ACTIVE",
    "new_status": "REVOKED",
    "name": "production",
    "cascade_reason": "tenant_closed"
  },
  "correlation_id": "tenant_close_cascade:acme-corp:req_...",
  "trace_id": "<same as originating>"
}
```

### `webhook.disabled_via_tenant_cascade`

Emitted once per owned `WebhookSubscription` when the tenant closes. Status flips to `DISABLED`; re-enable is blocked by the Rule 2 guard (returns `409 TENANT_CLOSED`), making DISABLED effectively-terminal for closed-owner subscriptions without adding a new enum value.

```json
{
  "event_id": "evt_...",
  "event_type": "webhook.disabled_via_tenant_cascade",
  "category": "webhook",
  "tenant_id": "acme-corp",
  "data": {
    "subscription_id": "whsub_...",
    "prior_status": "ACTIVE",
    "new_status": "DISABLED",
    "name": "ops-alerts",
    "cascade_reason": "tenant_closed"
  },
  "correlation_id": "tenant_close_cascade:acme-corp:req_...",
  "trace_id": "<same as originating>"
}
```

### Correlating cascade events

The shared `correlation_id` is the primary join key — querying `GET /v1/admin/events?correlation_id=...` returns every event emitted by the cascade in one call. The dashboard (v0.1.25.43+) renders a "tenant cascade" chip on audit and event-timeline rows with these suffixes. See [Using the Cycles Dashboard](/how-to/using-the-cycles-dashboard#closed-tenant-tombstone-and-cascade-preview).

**No emission-order guarantee.** The spec's ordering language covers the cascade's *mutations* (a SHOULD, and only within Mode A's single transaction — see [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics#the-two-rules)); it is silent on cascade *event emission* order. The reference implementation currently emits per budget — `budget.closed_via_tenant_cascade` then, for budgets with `reserved > 0`, the `reservation.released_via_tenant_cascade` aggregate, interleaved budget by budget — followed by webhook and API-key events, but this is implementation detail. Subscribers MUST NOT rely on arrival order (at-least-once webhook delivery can reorder and duplicate regardless — see [delivery mechanics](/protocol/webhook-event-delivery-protocol)); reconstruct the cascade by joining on the shared `correlation_id` instead.

---

## Webhook Lifecycle Events

**Currently emitted (spec v0.1.25.33).** Admin v0.1.25.39 emits six webhook lifecycle event types on the subscription CRUD + bulk-action paths; events v0.1.25.11 emits `webhook.disabled` on the dispatcher auto-disable path. All six share the `EventDataWebhookLifecycle` payload and the `webhook` category.

### `EventDataWebhookLifecycle` payload

| Field | Type | Always present | Description |
|---|---|---|---|
| `subscription_id` | string | Yes | The affected webhook subscription (`whsub_...`). |
| `tenant_id` | string | Yes | Owning tenant — mirrors the envelope for convenience. |
| `previous_status` | string | When applicable | `ACTIVE` / `PAUSED` / `DISABLED`. Absent on `webhook.created` (no prior state). Present on `webhook.deleted` (the status the subscription held before deletion). |
| `new_status` | string | When applicable | `ACTIVE` / `PAUSED` / `DISABLED`. Post-mutation status. Absent on `webhook.deleted` (subscription no longer exists). |
| `changed_fields` | array&lt;string&gt; | On `webhook.updated` | The subscription fields the PATCH actually modified (diff vs prior snapshot — identity-PATCHes emit an empty array and full-identity PATCHes suppress emit entirely per spec §6281). |
| `disable_reason` | string | On `webhook.disabled` | Why the dispatcher auto-disabled this subscription. Canonical value: `consecutive_failures_exceeded_threshold`. |

### `webhook.created`

**Trigger:** Successful `POST /v1/admin/webhooks`.
**Emitted by:** `cycles-server-admin` v0.1.25.39.
**Correlation-id shape:** `webhook_create:<subscription_id>`.

```json
{
  "event_id": "evt_...",
  "event_type": "webhook.created",
  "category": "webhook",
  "tenant_id": "acme-corp",
  "source": "cycles-admin",
  "data": {
    "subscription_id": "whsub_...",
    "tenant_id": "acme-corp",
    "new_status": "ACTIVE"
  },
  "correlation_id": "webhook_create:whsub_...",
  "trace_id": "<32-hex>"
}
```

### `webhook.updated`

**Trigger:** `PATCH /v1/admin/webhooks/{id}` that is neither a pure `ACTIVE → PAUSED` nor `PAUSED → ACTIVE` flip (those emit `webhook.paused` / `webhook.resumed` instead).
**Emitted by:** `cycles-server-admin` v0.1.25.39.
**Correlation-id shape:** `webhook_update:<subscription_id>:<request_id>`.

`changed_fields` is a true diff against the prior snapshot: re-PATCHing the same values is silently suppressed — no event emitted — so operators don't see lifecycle noise from identity writes.

### `webhook.paused`

**Trigger:** `PATCH /v1/admin/webhooks/{id}` with a status transition `ACTIVE → PAUSED`, or `POST /v1/admin/webhooks/bulk-action` with `action=PAUSE`.
**Emitted by:** `cycles-server-admin` v0.1.25.39.
**Correlation-id shape:** `webhook_update:<id>:<request_id>` (single-op) or `webhook_bulk_action:pause:<request_id>` (bulk).

### `webhook.resumed`

**Trigger:** `PATCH /v1/admin/webhooks/{id}` with a status transition `PAUSED → ACTIVE`, or `POST /v1/admin/webhooks/bulk-action` with `action=RESUME`.
**Emitted by:** `cycles-server-admin` v0.1.25.39.
**Correlation-id shape:** `webhook_update:<id>:<request_id>` (single-op) or `webhook_bulk_action:resume:<request_id>` (bulk).

### `webhook.disabled`

**Trigger:** The dispatcher auto-disables a subscription after consecutive delivery failures cross `disable_after_failures`.
**Emitted by:** `cycles-server-events` v0.1.25.11.
**Correlation-id shape:** `webhook_auto_disable:<subscription_id>:<delivery_id>`.
**Actor:** `{type: system}` with `source = cycles-events`.

This is reserved for dispatcher-driven disables. Operator-initiated disables show up as `webhook.paused` (soft-disable) or `webhook.deleted` (removal). Tenant-close cascades use the separate `webhook.disabled_via_tenant_cascade` event — see [Tenant-Close Cascade Events](#webhook-disabled-via-tenant-cascade) above.

```json
{
  "event_id": "evt_...",
  "event_type": "webhook.disabled",
  "category": "webhook",
  "tenant_id": "acme-corp",
  "source": "cycles-events",
  "actor": { "type": "system" },
  "data": {
    "subscription_id": "whsub_...",
    "tenant_id": "acme-corp",
    "previous_status": "ACTIVE",
    "new_status": "DISABLED",
    "disable_reason": "consecutive_failures_exceeded_threshold"
  },
  "correlation_id": "webhook_auto_disable:whsub_...:dlv_...",
  "trace_id": "<copied from triggering delivery when present>"
}
```

### `webhook.deleted`

**Trigger:** Successful `DELETE /v1/admin/webhooks/{id}`, or `POST /v1/admin/webhooks/bulk-action` with `action=DELETE`.
**Emitted by:** `cycles-server-admin` v0.1.25.39.
**Correlation-id shape:** `webhook_delete:<subscription_id>` (single-op) or `webhook_bulk_action:delete:<request_id>` (bulk).

### Correlating webhook lifecycle events

Bulk-action invocations stamp every per-row emit with a shared `correlation_id` (`webhook_bulk_action:<action>:<request_id>`) — query `GET /v1/admin/events?correlation_id=...` to pull every lifecycle event from one operator action. Skipped or failed rows never emit. See [Using Bulk Actions](/how-to/using-bulk-actions-for-tenants-and-webhooks) for the full bulk-action event contract.

For the managing-webhooks operator flow (subscription creation, signing-secret rotation, delivery health) see [Managing Webhooks](/how-to/managing-webhooks).

---

## Tenant, API Key, Policy, and System Events

Current services emit part of every category below. The tables distinguish direct lifecycle emission, synthetic test delivery, and values that remain registered-but-planned.

### Tenant Events (6 types — 5 currently emitted)

| Event Type | Status | Trigger |
|---|---|---|
| `tenant.created` | **Emitted** (current admin implementation) | New tenant provisioned |
| `tenant.updated` | **Emitted** (current admin implementation) | Tenant configuration changed |
| `tenant.suspended` | **Emitted** (admin v0.1.25.38+) | `PATCH /v1/admin/tenants/{id}` or `bulk-action` sets status to `SUSPENDED` |
| `tenant.reactivated` | **Emitted** (admin v0.1.25.38+) | `PATCH /v1/admin/tenants/{id}` or `bulk-action` restores status to `ACTIVE` |
| `tenant.closed` | **Emitted** (admin v0.1.25.38+) | `PATCH /v1/admin/tenants/{id}` or `bulk-action` sets status to `CLOSED` — also triggers the four `_via_tenant_cascade` events documented above |
| `tenant.settings_changed` | Planned | Tenant default settings modified |

### API Key Events (6 base types, plus the tenant-cascade event)

| Event Type | Status | Trigger |
|---|---|---|
| `api_key.created` | **Emitted** (admin v0.1.25+) | New API key generated |
| `api_key.revoked` | **Emitted** (admin v0.1.25+) | API key permanently revoked |
| `api_key.expired` | Planned | API key reached expiration date |
| `api_key.permissions_changed` | **Emitted** (admin v0.1.25.7+) | API key permissions modified |
| `api_key.auth_failed` | **Emitted** (admin v0.1.25+) | Authentication attempt failed |
| `api_key.auth_failure_rate_spike` | Planned | Auth failure rate exceeded threshold |

`api_key.revoked_via_tenant_cascade` is emitted separately by admin v0.1.25.35+ when tenant close revokes owned keys.

### Policy Events (3 types — 2 currently emitted)

| Event Type | Status | Trigger |
|---|---|---|
| `policy.created` | **Emitted** (admin v0.1.25+) | New policy rule created |
| `policy.updated` | **Emitted** (admin v0.1.25+) | Policy configuration changed |
| `policy.deleted` | Planned | Policy removed |

### System Events (5 types — 2 currently produced)

| Event Type | Status | Trigger |
|---|---|---|
| `system.store_connection_lost` | Planned | Redis connection failed |
| `system.store_connection_restored` | Planned | Redis connection recovered |
| `system.high_latency` | Planned | Server-side p99 latency exceeded threshold |
| `system.webhook_delivery_failed` | **Emitted** (events v0.1.25.21+) | Webhook delivery permanently failed after all retries; persisted as a loop-safe meta-event rather than recursively delivered |
| `system.webhook_test` | **Sent directly** (current admin implementation) | Admin-initiated synthetic connectivity test; not queued as an ordinary stored event |

---

## Event Emission Summary

| Category | Total Defined | Currently Emitted | Notes |
|---|---|---|---|
| Reservation | 6 | `reservation.denied`, `reservation.expired`, and `reservation.commit_overage` emitted by runtime paths; the cascade aggregate (`reservation.released_via_tenant_cascade`, spec-declared since governance v0.1.25.35) emitted by the admin server on tenant close | Spike events still planned |
| Budget | 17 | Runtime exhaustion/over-limit/debt events; admin create/update/freeze/unfreeze/funding events; `budget.closed_via_tenant_cascade` on tenant close | `budget.closed`, `budget.threshold_crossed`, `budget.over_limit_exited`, and `budget.burn_rate_anomaly` are not emitted |
| Tenant | 6 | `tenant.created`, `tenant.updated`, `tenant.suspended`, `tenant.reactivated`, and `tenant.closed` emitted by the current admin service | `tenant.settings_changed` still planned |
| API Key | 7 | `api_key.created`, `.revoked`, `.permissions_changed`, `.auth_failed`, plus `.revoked_via_tenant_cascade` | Expiry and rate-spike events still planned |
| Policy | 3 | `policy.created`, `policy.updated` | `policy.deleted` still planned |
| Webhook | 7 | 6 lifecycle events (`webhook.created` / `.updated` / `.paused` / `.resumed` / `.disabled` / `.deleted`) from admin v0.1.25.39 + events v0.1.25.11, plus `webhook.disabled_via_tenant_cascade` (spec-declared since v0.1.25.35) on tenant close | All registered enum values emitted |
| System | 5 | `system.webhook_delivery_failed` persisted by the events service; `system.webhook_test` delivered directly by the admin test endpoint | Store-connection and high-latency events still planned |
| **Total** | **51** | See category rows above | — |

For webhook delivery mechanics, retry schedule, and signature verification, see the [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol).

For integration examples (PagerDuty, Slack, ServiceNow), see [Webhook Integrations](/how-to/webhook-integrations).



# How Decide Works in Cycles: Preflight Budget Checks Without Reservation

Sometimes a system needs to know whether an action would be allowed before committing to it.

Not to reserve budget.
Not to begin execution.
Just to ask: **is there room?**

That is what the decide endpoint does.

## What decide is

`POST /v1/decide` evaluates a budget request against current scope balances and returns a decision — without creating a reservation or modifying any budget state.

It is a read-only preflight check.

The response tells you:

- **ALLOW** — sufficient budget exists
- **ALLOW_WITH_CAPS** — sufficient budget exists, but soft constraints apply
- **DENY** — insufficient budget or policy block

No budget is held. No reservation is created. No commit or release is needed afterward.

## When to use decide

Decide is useful when the system needs budget awareness without budget commitment.

### 1. UI gating

Before showing an action button to a user, the frontend can check whether the action would be allowed under current budget.

If the answer is DENY, the UI can disable the button, show a warning, or suggest a cheaper alternative.

### 2. Planning and routing

An agent or orchestrator may need to decide between multiple possible actions based on budget availability.

Decide lets the system evaluate options without locking budget for all of them.

### 3. Pre-validation before expensive setup

Some workflows require setup steps before the actual model or tool call. Decide lets the system check budget feasibility before investing in that setup.

### 4. Soft-landing signals

The decide endpoint can return caps (via ALLOW_WITH_CAPS) that signal the system should adjust its behavior — use fewer tokens, avoid certain tools, or slow down — without denying execution outright.

### 5. Dashboard and operator visibility

Operators can use decide to evaluate hypothetical scenarios against live budget state without affecting production accounting.

## How decide works in the protocol

A decide request includes four required fields:

- **subject** — the six standard fields that derive budget scopes (tenant, workspace, app, workflow, agent, toolset), plus optional `dimensions` metadata that does not derive scopes in the reference server
- **action** — the proposed action (kind, name, optional tags)
- **estimate** — the amount the action would need
- **idempotency_key** — for request deduplication

An optional `metadata` map can be attached for application-level context.

The server evaluates the request against current balances and returns:

- **decision** — ALLOW, ALLOW_WITH_CAPS, or DENY
- **caps** — soft constraints (only present when decision is ALLOW_WITH_CAPS)
- **reason_code** — machine-readable reason when decision is DENY (`DecisionReasonCode`, see below)
- **retry_after_ms** — optional guidance on when to retry
- **affected_scopes** — which scopes were evaluated
- **cycles_evidence** — a reference to the signed CyclesEvidence envelope emitted for this decision (artifact type `decide`; present on ALLOW, ALLOW_WITH_CAPS, and DENY outcomes, absent only when evidence emission is disabled on the server)

Only `decision` is required in the response — every other field is conditional or optional.

`DecisionReasonCode` is an **open string** (not a closed enum) with the following documented known values:

| reason_code | Meaning |
|---|---|
| `BUDGET_EXCEEDED` | Remaining amount insufficient on at least one derived scope |
| `BUDGET_FROZEN` | A derived scope has a budget in `FROZEN` status |
| `BUDGET_CLOSED` | A derived scope has a budget in `CLOSED` status |
| `BUDGET_NOT_FOUND` | No budget exists at any derived scope in the requested unit (on non-dry reserve and `/v1/events`, this same condition surfaces as `HTTP 404` with `error=NOT_FOUND`) |
| `OVERDRAFT_LIMIT_EXCEEDED` | Either `debt + delta > overdraft_limit`, or the scope is in over-limit state (`is_over_limit=true`) |
| `DEBT_OUTSTANDING` | A derived scope has `debt > 0` and `overdraft_limit == 0` |
| `TENANT_CLOSED` | The owning tenant's status is `CLOSED` (deployments with a governance plane; added in spec v0.1.25.13, emitted by cycles-server 0.1.25.47+ on fresh evaluations — the persisting reservation mutations surface the same condition as `HTTP 409` with `error=TENANT_CLOSED`) |

`DecisionReasonCode` was widened from a closed enum to an open string in v0.1.25 so future extension specs can add new reason codes without a breaking protocol bump. **Clients MUST handle unknown values gracefully** (treat as DENY, log the raw string, do not crash on enum parsing). Known values above are stable; future values will always be additive. See [Decision reason codes](/protocol/error-codes-and-error-handling-in-cycles#decision-reason-codes) for full semantics.

The v0.1.26 runtime extension adds three action-governance reason codes that can surface on decide denials:

| reason_code | Meaning |
|---|---|
| `ACTION_QUOTA_EXCEEDED` | A per-kind or risk-class action quota rule was exceeded for the target scope and window |
| `ACTION_KIND_DENIED` | The action kind is in the matching policy's `denied_action_kinds` list |
| `ACTION_KIND_NOT_ALLOWED` | The matching policy has a non-empty `allowed_action_kinds` list and the action kind is not in it |

See `cycles-protocol-extensions-v0.1.26.yaml` for full semantics, evaluation order, and the `DenyDetail` structure populated alongside `reason_code` for these denials.

## Decide does not guarantee future reservation

An important subtlety: decide is a point-in-time evaluation.

If decide returns ALLOW at time T, a subsequent reservation at time T+1 may still fail if concurrent activity consumed budget in between.

Decide is advisory. It reflects current state, not a promise about future state.

For guaranteed budget holds, use reservations.

## Decide and debt/overdraft

When a scope has outstanding debt or is in over-limit state, the server SHOULD return `DENY` with the appropriate reason code (`DEBT_OUTSTANDING` or `OVERDRAFT_LIMIT_EXCEEDED`). This is a SHOULD, not a MUST — but the server MUST NOT return `409` for these conditions on decide.

For budget-state conditions — debt, overdraft, insufficient remaining — decide returns a `200` response with a decision value rather than a 4xx error. The exception is request-validity errors: if the estimate's unit doesn't match any budget at the derived scopes (but at least one of those scopes has a budget in a different unit), decide returns `400 UNIT_MISMATCH` just like the other endpoints.

This makes decide safe to call in any context without needing error handling for budget state issues.

## Decide vs dry_run

Both decide and `dry_run: true` on a reservation request evaluate budget without modifying state. But they serve different purposes:

- **decide** is a lightweight check that returns a decision and optional caps
- **dry_run** evaluates the full reservation path including scope derivation, affected scopes, and balance snapshots

Use decide for quick feasibility checks. Use dry_run for full shadow-mode evaluation of reservation logic.

## A practical example

An agent is planning its next step. It has two options:

1. Call a large model (estimated 8,000 tokens)
2. Call a smaller model (estimated 2,000 tokens)

Before choosing, the agent calls decide for each option:

- Option 1: decide returns DENY (run budget is too low)
- Option 2: decide returns ALLOW

The agent routes to the smaller model without ever creating a reservation for the larger one.

This avoids wasting budget on reservation overhead for paths that would be denied.

## When not to use decide

Decide is not a substitute for reservations.

If the system needs:

- guaranteed budget holds before execution
- concurrency-safe budget enforcement
- commit/release lifecycle tracking

then create a reservation instead.

Decide is for asking questions. Reservations are for taking action.

## Summary

Decide provides a read-only preflight check against live budget state.

It helps systems:

- gate actions before committing to them
- route between alternatives based on budget availability
- check feasibility without locking budget
- receive soft-landing signals via caps

It is advisory, not binding. For guaranteed budget holds, use reservations.

## Next steps

To explore the Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)



# How Events Work in Cycles: Direct Debit Without Reservation

The core Cycles lifecycle is reserve → execute → commit.

That pattern works well when the system can estimate exposure before execution, hold bounded room, and reconcile actual usage afterward.

But not every action fits that pattern.

Sometimes the cost is already known.
Sometimes the work already happened.
Sometimes reservation overhead is not justified.
Sometimes the system is recording usage from an external source that Cycles does not control.

That is where events come in.

## What events are

An event in Cycles is a direct debit against a budget scope — without creating a reservation first.

The API endpoint is `POST /v1/events`.

Instead of:

1. reserve
2. execute
3. commit

an event does:

1. record actual usage directly

The server applies the usage atomically across all derived scopes, just like a commit would, but without the preceding reservation step.

## When to use events

Events are useful when:

- the cost is already known before the call
- the action already completed outside of Cycles
- reservation overhead is not justified for the action class
- the system is importing historical or external usage into the budget ledger
- the action is low-risk and does not need pre-execution budget authorization

### Examples

- recording a model call that already happened through an external gateway
- importing usage from a billing provider into Cycles for unified budget tracking
- logging a known-cost action like sending an email or creating a ticket
- accounting for background work that was not instrumented with reserve-commit
- migrating from a legacy usage system into Cycles

## When not to use events

Events should not replace reservations for actions where pre-execution budget control matters.

If the system needs to:

- decide whether work is allowed before it runs
- hold bounded room before expensive execution
- protect against concurrent over-consumption
- enforce run-level or workflow-level budget ceilings proactively

then reserve → commit is the right pattern.

Events are post-hoc accounting. Reservations are pre-execution control.

Both are useful. They solve different problems.

## How events work in the protocol

An event request includes:

- **subject** — the six standard fields that derive budget scopes (tenant, workspace, app, workflow, agent, toolset), plus optional `dimensions` metadata that does not derive scopes in the reference server
- **action** — what happened (kind, name, optional tags)
- **actual** — the amount consumed (unit and amount)
- **idempotency_key** — ensures the same event is not recorded twice
- **overage_policy** — what happens if budget is insufficient (REJECT, ALLOW_IF_AVAILABLE, or ALLOW_WITH_OVERDRAFT)
- **metrics** — optional operational metadata (tokens_input, tokens_output, latency_ms, model_version, custom)
- **client_time_ms** — optional client-observed timestamp (advisory only, not used for budget enforcement)
- **metadata** — optional arbitrary key-value metadata for audit or debugging

The server applies the charge atomically across all derived scopes, or rejects the entire event.

On success, the server returns `201` — the event has been created and atomically applied to balances before the response is sent. The response includes:

- `status: APPLIED`
- `event_id` — a unique identifier for the event
- `charged` — the amount actually applied (optional; present when `ALLOW_IF_AVAILABLE` caps the charge to remaining budget, so the client can see the effective charge)
- `balances` — updated balance state for affected scopes

### Error cases

Beyond the `409` overage rejections described below, the spec defines these error responses for `POST /v1/events`:

- `400 UNIT_MISMATCH` — the actual's unit doesn't match the budget stored for the target scope (a budget exists at the scope, but in a different unit)
- `404 NOT_FOUND` — no budget exists at any of the event's derived scopes in *any* unit; the message field carries the specific `"Budget not found for provided scope: ..."` detail
- `403 FORBIDDEN` — the request's `subject.tenant` doesn't match the effective tenant derived from auth
- `409 IDEMPOTENCY_MISMATCH` — the same idempotency key was reused with a different payload (see Idempotency below)

## Overage policies on events

Events support the same three overage policies as commits:

### REJECT

If the actual amount exceeds the available budget, the event is rejected with `409 BUDGET_EXCEEDED`.

REJECT prevents any accounting that would put the scope into negative remaining.

### ALLOW_IF_AVAILABLE (default)

If sufficient budget remains across all affected scopes, the full actual amount is applied atomically. If any scope has insufficient remaining, the charge is capped to available remaining (the minimum across all derived scopes, floor 0) and `is_over_limit` is set to `true` only on the scopes where the full amount could not be covered.

ALLOW_IF_AVAILABLE is the default when the request does not specify a policy. It never creates debt and never rejects an event.

### ALLOW_WITH_OVERDRAFT

If budget is insufficient, the system creates debt up to the scope's overdraft limit. If debt would exceed the overdraft limit, the event is rejected with `409 OVERDRAFT_LIMIT_EXCEEDED`.

This is useful for ensuring the ledger always reflects reality, even when budget is tight.

## Idempotency

Events are idempotent.

If the same `idempotency_key` is sent again with the same payload, the server returns the original response without applying the charge a second time.

If the same key is sent with a different payload, the server returns `409 IDEMPOTENCY_MISMATCH`.

This makes events safe under retries.

## Events vs reservations

| | Reservations | Events |
|---|---|---|
| Pre-execution control | Yes | No |
| Budget held before work | Yes | No |
| Commit/release lifecycle | Yes | No |
| Idempotent | Yes | Yes |
| Scope derivation | Yes | Yes |
| Overage policies | Yes | Yes |
| Best for | Actions with uncertain cost | Actions with known cost |

## A practical example

Suppose an external gateway processes a model call and reports back that it consumed 4,200 tokens.

The system can record this in Cycles with a single event:

- subject: tenant `acme`, app `support-bot`
- action: kind `llm.completion`, name `openai:gpt-4o-mini`
- actual: 4200 TOKENS
- overage_policy: ALLOW_IF_AVAILABLE

The server applies this charge across all derived scopes (tenant, app) and returns the updated balances.

No reservation was needed because the work already happened and the cost is already known.

## When to combine events with reservations

Many systems use both patterns.

For example:

- **reservations** for model calls and tool invocations that the system controls directly
- **events** for external usage imports, historical data, or low-cost background work

This gives the system pre-execution control where it matters most, and simple accounting everywhere else.

## Summary

Events provide a way to record known usage directly against budget scopes without the overhead of a reservation lifecycle.

They are useful for:

- known-cost actions
- external usage imports
- retroactive accounting
- low-risk actions where reservation is not justified

They support the same overage policies, idempotency, and scope derivation as reservations.

The key difference: reservations authorize work before it happens. Events record work after it happens.

Both are part of a complete Cycles deployment.

## Next steps

To explore the Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)



# How Reserve → Commit Works in Cycles

Most systems discover usage after work is already done.

A model call finishes.  
A tool runs.  
A database is written.  
A provider bill updates later.  
A dashboard shows the damage after the fact.

That is observability.

It is not control.

Cycles uses a different model:

1. **Reserve exposure before execution**
2. **Execute the work**
3. **Commit actual usage or release the remainder**

This is the core execution pattern in Cycles.

It is what makes budget control work in systems with retries, loops, concurrency, and long-running workflows.

## Why reserve first?

In autonomous systems, cost and side effects are often created by software that keeps acting after the initial request is gone.

An agent may:

- call multiple models
- invoke tools recursively
- retry on failure
- fan out into parallel steps
- continue in the background

If you only measure usage after these actions complete, you are not really governing execution.

You are auditing it after the fact.

Reservation changes that.

Before work starts, the system asks:

::: info
Is this action allowed to consume up to this amount of exposure?
:::

If the answer is no, the action can be denied, downgraded, deferred, or rerouted before cost or side effects occur.

## The basic lifecycle

At a high level, Cycles follows this pattern:

### 1. Declare intent

An action identifies the scope and the estimated amount of exposure it may need.

That may correspond to:

- a model call
- a tool invocation
- a workflow step
- an agent run
- any other governable action

### 2. Reserve budget

The runtime attempts to reserve that amount against the relevant budget scopes.

If reservation succeeds, the action is allowed to proceed.

If reservation fails, the system can stop or degrade the action before execution.

### 3. Execute

The work runs.

At this point, the system knows it is operating within a bounded allowance.

### 4. Commit actual usage or release

Once the real usage is known, the runtime commits the actual amount consumed.

If the actual amount is lower than the reserved amount, the unused portion is released automatically as part of the commit. No separate release call is needed.

If work is canceled or fails before any usage occurs, the runtime releases the reservation explicitly, returning the full reserved amount to the budget pool.

::: tip SDK behavior
The official Python `@cycles`, TypeScript `withCycles`, Spring `@Cycles`, and Rust `ReservationGuard` helpers automate this lifecycle. When actual usage is known, current releases persist settlement before the first commit request, reuse the same idempotency key after ambiguous outcomes, and recover an expired commit through `POST /v1/events`. See [SDK Settlement Recovery and Durability](/protocol/sdk-settlement-recovery-and-durability).
:::

## A simple example

Suppose an agent wants to run a tool-assisted task and estimates it may consume 100 units.

The lifecycle looks like this:

1. Reserve 100
2. Execute the task
3. Actual usage ends up being 63
4. Commit 63 (the remaining 37 is released automatically)

Without this model, many systems do one of two bad things:

- they block based on rough static quotas
- they allow execution freely and reconcile later

Cycles allows bounded execution with reconciliation afterward.

## Why commit matters

Reservation alone is not enough.

If you reserve budget and never reconcile actual usage, your system becomes inaccurate very quickly.

You need commit because estimated cost and actual cost are often different.

For example:

- a model call returns sooner than expected
- a tool path exits early
- a workflow skips downstream actions
- a retry path consumes more than the optimistic estimate

Commit turns reserved intent into actual accounted usage.

That keeps balances meaningful.

## Why release matters

Release is just as important as commit.

If you reserve exposure and do not return unused portions, the system slowly accumulates phantom consumption.

That causes two problems:

- budgets appear tighter than they really are
- future work is denied unnecessarily

When actual usage is committed, the protocol automatically releases the unused remainder. When work is canceled or fails before any usage occurs, the runtime must explicitly release the reservation.

Explicit release is especially important for:

- canceled jobs
- partially completed workflows where no usage should be recorded
- guarded speculative execution
- early exits
- timeout handling

## Reserve → commit is different from post-hoc usage tracking

A lot of systems already track usage.

That is not the same thing.

Post-hoc tracking says:

::: info
We can tell you what happened after execution.
:::

Reserve → commit says:

::: info
We decide whether work may proceed before execution, then reconcile actual usage afterward.
:::

That difference is the entire point.

The first model supports reporting.  
The second supports governance.

## Reserve → commit is different from flat quotas

Flat quotas are useful, but limited.

A quota might say:

- tenant A can spend 10,000 units today
- workflow B can use 500 units per run

That helps at the policy level.

But execution still needs a transactional pattern.

Reserve → commit provides that operational layer.

It answers questions like:

- can this action proceed right now?
- how much room has already been set aside?
- how do we avoid double counting?
- what happens if execution fails halfway through?
- how do we reconcile estimates with actuals?

## Why this matters under retries

Retries are one of the main reasons Cycles exists.

In real systems, retries happen because of:

- transient provider errors
- network instability
- worker crashes
- downstream timeouts
- optimistic retry logic

Without careful accounting, retries often create double-spend or hidden overages.

A reserve → commit model, combined with idempotency, gives the runtime a way to keep retries safe.

Instead of each retry being treated as brand-new spend, the system can reason about the same execution lifecycle consistently.

That is how you move from “best effort cost control” to deterministic budget enforcement.

## Why this matters under concurrency

Concurrency makes naive accounting unreliable.

Two workers can both believe budget is available.  
Two steps can race.  
A parent workflow and child tasks can all consume at once.

If accounting is not designed for this, you get inconsistent enforcement, accidental overages, or brittle locking.

Cycles is built around reservation semantics so the system can make budget decisions before work proceeds, rather than discovering collisions only after usage has already occurred.

## Hierarchical reservation

In Cycles, reservation is not limited to one flat counter.

An action may need to satisfy multiple scopes at once, such as:

- tenant
- workspace
- app
- workflow
- agent
- toolset

That means a reservation may need to be valid not only locally, but also against ancestor scopes.

This is important because real systems often need both:

- a global budget boundary
- more specific per-run or per-agent boundaries

Reserve → commit works cleanly with hierarchical governance because the runtime can check and enforce multiple levels before execution begins.

## What happens when reservation fails?

A failed reservation is not just a denial event.

It is a policy decision point.

Depending on the system, failure may result in:

- hard stop
- retry later
- downgrade to a smaller model
- disable expensive tools
- switch to read-only behavior
- reduce concurrency
- require operator approval

This is another reason reserve → commit is powerful.

It creates a place to make a bounded decision before irreversible work begins.

## What happens when work crashes?

Real workflows fail halfway through.

A worker may crash after reservation but before commit.  
A process may be terminated.  
A network partition may delay reconciliation.

A usable budget system has to handle these cases explicitly.

That is why Cycles treats reserve, commit, and release as separate lifecycle events rather than assuming execution is always clean and synchronous.

Current SDK lifecycle helpers durably preserve settlement once actual usage is known. They write the unresolved settlement before its first commit request, replay ambiguous outcomes with the original idempotency key, and use a direct event if the reservation expires before commit can land.

That guarantee has a hard boundary: if the process dies before the downstream operation returns an actual amount, the SDK has no amount to recover. Applications that require convergence across that boundary must durably checkpoint provider receipts or actual usage before acknowledging the downstream operation.

## Mental model

A simple way to think about it is:

- **Reserve** = hold bounded room to act
- **Commit** = record what was actually consumed (auto-releases unused remainder)
- **Release** = cancel the reservation and return what was held

That is the core accounting discipline.

Without reserve, budgets are advisory.  
Without commit, estimates drift from reality.  
Without release, unused allocations become silent waste.

## Why this model matters

Reserve → commit is not extra ceremony.

It is the difference between:

- hoping autonomous systems stay within budget
- and making budget enforcement part of execution itself

As systems become more autonomous, they need more than request throttling and usage dashboards.

They need a runtime model that can:

- authorize bounded work before execution
- keep accounting stable under retries
- reconcile actual usage afterward
- enforce policy across scopes
- degrade safely when budgets are tight

That is what Cycles provides.

## Next steps

To see where this model is defined and implemented:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)



# How Scope Derivation Works in Cycles

Budget enforcement in Cycles is hierarchical.

A single action does not just charge one counter. It may need to satisfy limits at multiple levels — tenant, workspace, app, workflow, agent, toolset — all at once.

That is what makes Cycles different from flat usage counters.

But how does the server know which budget scopes to check and charge?

That is scope derivation.

## What scope derivation is

Scope derivation is the process by which the server transforms a Subject (a bag of dimension fields) into an ordered set of canonical scope identifiers.

Derivation happens wherever a Subject is submitted: reserve (`POST /v1/reservations`), decide (`POST /v1/decide`), and event (`POST /v1/events`). When one of those requests is processed, the server derives every scope that needs to be affected. Commit and release do not take a Subject — they operate on the scopes already derived at reserve time.

For example, given a Subject with:

- tenant: `acme`
- app: `support-bot`
- workflow: `refund-assistant`

The server derives these scopes (in canonical order):

1. `tenant:acme`
2. `tenant:acme/app:support-bot`
3. `tenant:acme/app:support-bot/workflow:refund-assistant`

Only explicitly provided levels are included — `workspace` is not present in the subject, so it is skipped. Each of these scopes is a separate budget boundary. A reservation is enforced at every derived scope that has a budget defined — at least one scope must have a budget.

## The canonical hierarchy

The Cycles protocol defines a fixed ordering for Subject fields:

```
tenant → workspace → app → workflow → agent → toolset
```

This ordering is normative. The server always processes fields in this order, and `affected_scopes` in responses are always returned in this canonical order.

## Gap-skipping

Not every Subject includes all six fields. When a field is missing from the hierarchy, the server skips it — only explicitly provided levels appear in the scope path.

For example, given:

- tenant: `acme`
- agent: `summarizer-v2`

The derived scope path is:

```
tenant:acme/agent:summarizer-v2
```

And the derived scopes are:

1. `tenant:acme`
2. `tenant:acme/agent:summarizer-v2`

Intermediate levels (`workspace`, `app`, `workflow`) are not present in the subject and are not filled with "default". This means operators only need to create budgets at levels they actually use, rather than at every intermediate level in the hierarchy.

Scopes without budgets are skipped during enforcement — at least one derived scope must have a budget defined.

## Field value charset

Standard-field values (tenant, workspace, app, workflow, agent, toolset) SHOULD match `^[a-zA-Z0-9_.-]+$` (normative as of the 2026-07-03 spec revision).

Canonical scope identifiers and scope paths use `:` and `/` as structural delimiters, so delimiter, whitespace, and control characters have no stable canonical encoding.

Servers MAY reject out-of-pattern values with `400 INVALID_REQUEST` — the reference implementation does. Portable clients SHOULD restrict themselves to the pattern above and MUST NOT rely on out-of-pattern values being accepted.

## Why hierarchical scopes matter

Hierarchical scopes enable layered budget governance.

### Tenant-level protection

The tenant scope ensures that all actions under a tenant count against the tenant's total budget.

No matter which app, workflow, or agent is running, the tenant boundary is always checked.

### App-level isolation

Different applications under the same tenant can have different budgets.

A support bot and a research agent can each have their own budget envelope, preventing one from consuming the other's allocation.

### Workflow-level control

Within an app, different workflows can have different cost profiles.

A refund workflow may justify more budget than a simple FAQ response.

### Agent-level boundaries

Within a workflow, individual agents can be bounded separately.

A planning agent and an execution agent can each have their own limits.

### Toolset-level restrictions

Within an agent, different toolsets can be governed independently.

Web search tools may have a different budget than database query tools.

## Atomic reservation across scopes

When a reservation is created, budget is reserved atomically across all derived scopes.

If any scope has insufficient budget, the reservation fails entirely. There is no partial reservation.

This prevents situations where a reservation succeeds at the local level but violates a parent boundary.

For example, if a workflow has budget remaining but the tenant is exhausted, the live reservation is rejected at the tenant scope.

## affected_scopes in responses

Reservation and decision responses include an `affected_scopes` field listing all scopes that were charged (or would be charged), in canonical order.

Event responses do not include `affected_scopes` — for `POST /v1/events`, read the charged scopes from the `scope_path` on each entry in the response's `balances` array instead.

Where present, `affected_scopes` tells the client exactly which budget boundaries were affected, which is useful for:

- debugging denial reasons
- understanding which scope is the bottleneck
- monitoring budget pressure across the hierarchy

## scope_path

The `scope_path` field in responses is the full canonical path for the reservation.

For example: `tenant:acme/app:support-bot/workflow:refund-assistant`

This uniquely identifies the leaf scope in the hierarchy.

## Custom dimensions

The Subject also supports a `dimensions` field for custom key-value pairs.

For example:

```json
{
  "tenant": "acme",
  "app": "support-bot",
  "dimensions": {
    "cost_center": "engineering",
    "run": "run-12345"
  }
}
```

In v0, servers MAY ignore dimensions for budgeting decisions entirely (the reference server does — scopes derive only from the six standard fields), but they must accept and round-trip the data.

Dimensions have hard limits: at most 16 keys, and each value at most 256 characters. Keys SHOULD be lowercase and match `^[a-z0-9_.-]+$` for stable canonicalization; values are opaque strings.

Dimensions never replace the standard fields. A Subject containing only `dimensions` — with no tenant, workspace, app, workflow, agent, or toolset — is invalid, and the server MUST return `400 INVALID_REQUEST`.

Dimensions are not budget-scope fields — they serve enterprise taxonomies, attribution/reporting, and policy uses (the v0.1.26 action-quota preview keys its `per_run` window off `dimensions.run_id`). To give each run an enforceable budget, encode the run id in a Subject field (e.g. `workflow: "run-{id}"`, deriving the scope `workflow:run-{id}`) — see [modeling tenant, workflow, and run budgets](/how-to/how-to-model-tenant-workflow-and-run-budgets-in-cycles).

## Scope derivation and balances

When querying balances (`GET /v1/balances`), the same scope hierarchy applies. At least one standard subject filter (tenant, workspace, app, workflow, agent, toolset) must be provided.

Returning child scopes is opt-in: pass `include_children=true` (the default is `false`). Be aware that `include_children` MAY be ignored by v0 implementations, so do not depend on child rows being present.

This gives operators visibility into budget state at any level of the hierarchy.

## Practical implications

### Budget allocation flows top-down

Tenant budgets constrain everything beneath them. If the tenant is exhausted, no child scope can reserve budget, even if the child has its own allocation.

### Pressure signals flow bottom-up

When a workflow or agent scope runs low, that pressure is visible at higher levels through balance queries and denial signals.

### Scope design is a policy decision

The scopes you populate on each Subject determine which budget boundaries are checked.

If you only provide tenant and workflow, the server derives scopes for tenant and workflow — intermediate levels are skipped. Agent and toolset scopes are not checked.

This means scope design is part of policy design. More scopes mean finer-grained control but more configuration.

## Summary

Scope derivation transforms Subject fields into a hierarchical set of budget boundaries:

- The canonical order is: tenant → workspace → app → workflow → agent → toolset
- Missing fields are skipped (not filled with "default")
- Reservations are atomic across all derived scopes
- Balances, affected_scopes, and scope_path all follow the same hierarchy
- Custom dimensions carry additional taxonomy and attribution metadata but do not derive budget scopes in the reference server

Understanding scope derivation is essential for designing effective budget policy in Cycles.

## Next steps

- [Tenants, Scopes, and Budgets](/how-to/understanding-tenants-scopes-and-budgets-in-cycles) — how the three building blocks fit together
- [Budget Allocation and Management](/how-to/budget-allocation-and-management-in-cycles) — create budgets at each scope level
- [Common Budget Patterns](/how-to/common-budget-patterns) — practical scope hierarchy recipes
- [Tenant, Workflow, and Run Budgets](/how-to/how-to-model-tenant-workflow-and-run-budgets-in-cycles) — multi-level policy design
- [Scope Misconfiguration and Budget Leaks](/incidents/scope-misconfiguration-and-budget-leaks) — what can go wrong with scope design



# Implement the Cycles Protocol

The Cycles Protocol is open. The reference server ([`runcycles/cycles-server`](https://github.com/runcycles/cycles-server)) is Apache 2.0 and validated against the spec, but it is not the only conformant implementation possible. Anyone can build a server that speaks the same wire format.

This page is the entry point for that work.

## Who this is for

- **Framework maintainers** — LangGraph, Temporal, agent runtimes, orchestrators that want first-class runtime budget authority without proxying every call to an external service.
- **Infrastructure teams** — operating an internal budget system already (custom rate limiter, cost tracker, action governance) and wanting Cycles-compatible client interop without replacing what works.
- **OSS contributors** — building adapters, alternative servers in other languages, or instrumented variants for specific deployment models (FaaS, edge, embedded).
- **Vendors with overlapping scope** — observability, cost tracking, identity governance vendors who want to add reserve-commit semantics to their existing platform.

If "should this specific next agent action proceed, given everything already consumed?" is a question your platform needs to answer, the protocol is the wire format for that question.

## Why a separate implementation might make sense

The reference Cycles server covers the canonical case: self-hosted, Java/Spring Boot, Redis-backed, multi-tenant. Reasons to implement separately:

- **Bespoke runtime requirements** — your stack already runs on Postgres, FoundationDB, DynamoDB, or an in-process state engine; you want budget authority co-located.
- **Integration with existing budget systems** — you've shipped a quota service for traditional API limits and want to expose it under the Cycles wire format so AI-agent SDKs work transparently.
- **Language / platform constraints** — embedded systems, FaaS edge runtimes, mobile / on-device agents where Java + Redis isn't viable.
- **Principled reasons** — you want a second independent implementation in the wild for protocol robustness (the OpenTelemetry multi-implementation pattern).

Cycles' reference server is fine; *protocol robustness* comes from multiple implementations testing the same wire format against the same conformance target — and, once published, the same conformance test kit.

## The minimum implementation surface

The authoritative statement of what's required lives in [`CONFORMANCE.md`](https://github.com/runcycles/cycles-protocol/blob/main/CONFORMANCE.md) — read that first. The current target requires a small core runtime + a cross-plane set, plus a smaller recommended set that well-rounded servers ship. Below is the surface at time of writing; if `CONFORMANCE.md` and this page disagree, **the spec wins**.

### Core runtime (MUST)

From [`cycles-protocol-v0.yaml`](https://github.com/runcycles/cycles-protocol/blob/main/cycles-protocol-v0.yaml):

| Operation | Endpoint | Purpose |
|---|---|---|
| **createReservation** | `POST /v1/reservations` | Atomically lock budget across all affected scopes before action |
| **commitReservation** | `POST /v1/reservations/{reservation_id}/commit` | Settle the actual cost; release the unused portion |
| **releaseReservation** | `POST /v1/reservations/{reservation_id}/release` | Release the full reservation without spending (cancel) |
| **extendReservation** | `POST /v1/reservations/{reservation_id}/extend` | TTL heartbeat for long-running operations |

### Cross-plane (MUST)

From [`cycles-governance-admin-v0.1.25.yaml`](https://github.com/runcycles/cycles-protocol/blob/main/cycles-governance-admin-v0.1.25.yaml). Much of the admin CRUD surface (tenant / budget / policy / key) is reference-shape rather than current conformance target, but the cross-plane operations below carry explicit `x-conformance: normative` labels and MUST follow the spec contract:

1. `GET /v1/admin/events` — **listEvents**
2. `GET /v1/admin/events/{event_id}` — **getEvent**
3. `POST /v1/admin/webhooks/{subscription_id}/replay` — **replayEvents**
4. `GET /v1/events` — **listTenantEvents** (tenant-scoped)
5. `GET /v1/admin/webhooks/{subscription_id}/deliveries` — **listWebhookDeliveries**
6. `GET /v1/webhooks/{subscription_id}/deliveries` — **listTenantWebhookDeliveries** (tenant-scoped)
7. `GET /v1/balances` — **getBalances** (admin-plane view)
8. `GET /v1/auth/introspect` — **introspectAuth**

### Recommended (SHOULD)

Well-rounded servers also implement these (from `cycles-protocol-v0.yaml`):

- `decide` — preflight budget check without reservation
- `listReservations` / `getReservation` — recovery and inspection
- `createEvent` — direct-debit event submission

You can ship without these and still claim conformance against the current target, but most clients expect them.

## Client recovery conformance

Server conformance is only one half of failure-safe settlement. SDK lifecycle helpers also need a shared contract for what happens when connectivity is lost after an action, the process restarts, a reservation expires before commit, or a heartbeat cannot be extended.

The authoritative [`client-recovery/PROFILE.md`](https://github.com/runcycles/cycles-protocol/blob/main/client-recovery/PROFILE.md) defines that client-side guarantee independently from the wire protocol. A durable-conformant SDK:

- persists known-actual settlement before the first request;
- reuses the same idempotency key after ambiguous outcomes;
- retains unresolved records across retry exhaustion, authentication failure, and restart;
- converts an expired commit into a same-key `POST /v1/events` settlement;
- exposes a bounded flush/drain operation; and
- makes heartbeat failure observable without suppressing final settlement.

The shared [`client-recovery/scenarios.yaml`](https://github.com/runcycles/cycles-protocol/blob/main/client-recovery/scenarios.yaml) catalog is executable. SDKs claiming durable recovery must bind every applicable scenario to native behavior tests and run the profile runner in CI. See [SDK Settlement Recovery and Durability](/protocol/sdk-settlement-recovery-and-durability) for the operational guarantee and configuration of the four official SDKs.

## The four core invariants

Spec compliance isn't just endpoint coverage — it's behavior under the hood. Per `CONFORMANCE.md`:

1. **Atomic reservation across scopes** — when a reservation locks multiple affected scopes, it locks all of them or none. No partial locks.
2. **Concurrency-safe enforcement** — shared budgets MUST NOT be oversubscribed under concurrent reserve calls. The reference server uses Lua-scripted Redis operations; alternative implementations need equivalent atomicity guarantees in their backing store.
3. **Idempotent commit and release** — every commit / release MUST be safe to retry. The same action MUST NOT settle twice. Idempotency keys carry the contract.
4. **Unit consistency** — reservations and commits MUST validate and preserve unit denomination (USD_MICROCENTS, TOKENS, CREDITS, RISK_POINTS). Cross-unit operations MUST return `UNIT_MISMATCH` (400).

These are the contracts client SDKs and downstream systems rely on. An implementation that returns the right HTTP codes but lets concurrent reservations oversubscribe is non-conformant in spirit even if it passes a naive endpoint test.

## Error semantics

Implementations MUST return the exact HTTP status + `error` code pairs from `cycles-protocol-v0.yaml` §ERROR SEMANTICS. The full set:

- `BUDGET_EXCEEDED` — 409
- `OVERDRAFT_LIMIT_EXCEEDED` — 409
- `IDEMPOTENCY_MISMATCH` — 409
- `RESERVATION_FINALIZED` — 409
- `RESERVATION_EXPIRED` — 410
- `UNIT_MISMATCH` — 400
- `NOT_FOUND` — 404
- `DEBT_OUTSTANDING` — 409
- `LIMIT_EXCEEDED` — 429 (server-side throttling / rate limiting, optional in v0 and never used for deterministic budget exhaustion; added to the ErrorCode enum in spec v0.1.25.12, revision 2026-07-04). 429 responses carry the `Retry-After` and `X-RateLimit-Reset` headers.
- `TENANT_CLOSED` — 409 (deployments with a governance plane only; added to the ErrorCode enum in spec v0.1.25.13, revision 2026-07-10, mirroring the governance code of the same name). Persisting reservation create/commit/release/extend MUST reject with it when the owning tenant's status is CLOSED and the flip is durable, taking precedence over `RESERVATION_FINALIZED`/`RESERVATION_EXPIRED` for non-replay attempts. Fresh `dry_run=true` and `/v1/decide` evaluations MUST NOT 409 for this — they return `200 decision=DENY reason_code=TENANT_CLOSED`; a present-but-malformed tenant record MUST fail closed with 500 `INTERNAL_ERROR`. Deployments with no tenant records anywhere are exempt (nothing to enforce).

One `RESERVATION_EXPIRED` subtlety, clarified in the 2026-07-03 spec revision: `GET /v1/reservations/{reservation_id}` MUST return 410 `RESERVATION_EXPIRED` when the reservation exists but its status is EXPIRED — the pre-revision text enumerated only commit/release/extend, leaving the GET case ambiguous. EXPIRED reservations remain discoverable via `listReservations`, which returns them as normal 200 rows with `status=EXPIRED`; the 410 applies only to the single-resource GET.

Clients route on these codes — returning a generic 500 or a custom error string breaks the protocol contract even if the underlying behavior is correct.

Action-governance error codes (`ACTION_QUOTA_EXCEEDED`, `ACTION_KIND_NOT_ALLOWED`, `ACTION_KIND_DENIED`) are documented in upcoming spec extensions; check `CONFORMANCE.md` for whether they're currently MUST or SHOULD against the active target.

## Authentication and tenancy

Authenticate via the `X-Cycles-API-Key` header, per [`cycles-protocol-v0.yaml`](https://github.com/runcycles/cycles-protocol/blob/main/cycles-protocol-v0.yaml) §AUTH & TENANCY. How API keys are provisioned, rotated, or scoped to permission sets is implementation-specific — but tenant isolation MUST be enforced.

Two exceptions are deliberately public (`security: []`): `GET /v1/evidence/{evidence_id}`, where the unguessable sha256 `evidence_id` acts as a capability, and `GET /v1/.well-known/cycles-jwks.json`, which serves the public verification-key set that is itself the trust anchor. Servers SHOULD rate-limit both (they declare 429 responses with `Retry-After` / `X-RateLimit-Reset`).

Three reservation operations additionally accept admin dual auth via `AdminKeyAuth` (`X-Admin-API-Key` header, the same header the governance-admin spec uses): `listReservations` (`GET /v1/reservations`), `getReservation` (`GET /v1/reservations/{reservation_id}`), and `releaseReservation` (`POST /v1/reservations/{reservation_id}/release`) — the operator use case is inspecting and force-releasing reservations across tenants, with the admin-driven release audited as `actor_type=admin_on_behalf_of`.

The reference server uses a permissions model with named scopes (`reservations:create`, `balances:read`, `admin:write`, etc.). Alternative implementations can use any equivalent authorization model as long as tenant isolation holds.

## Reference points

When you're stuck on a spec question, these are the canonical sources:

- **[`CONFORMANCE.md`](https://github.com/runcycles/cycles-protocol/blob/main/CONFORMANCE.md)** — the authoritative MUST / SHOULD / MAY document
- **[`cycles-protocol-v0.yaml`](https://github.com/runcycles/cycles-protocol/blob/main/cycles-protocol-v0.yaml)** — runtime base spec
- **[`cycles-spec-index.yaml`](https://github.com/runcycles/cycles-protocol/blob/main/cycles-spec-index.yaml)** — index of all spec files with conformance metadata
- **[`client-recovery/PROFILE.md`](https://github.com/runcycles/cycles-protocol/blob/main/client-recovery/PROFILE.md)** — client-side durable settlement and heartbeat-failure requirements
- **[`client-recovery/scenarios.yaml`](https://github.com/runcycles/cycles-protocol/blob/main/client-recovery/scenarios.yaml)** — executable recovery behavior catalog
- **[Reference server source](https://github.com/runcycles/cycles-server)** — the Java/Spring Boot reference implementation. Read it for "how does the reference handle X edge case?"
- **[Protocol reference pages](/protocol/api-reference-for-the-cycles-protocol)** — narrative documentation of the same surface, useful for understanding the design intent behind each operation
- **[`.spectral.yaml`](https://github.com/runcycles/cycles-protocol/blob/main/.spectral.yaml)** — OpenAPI linting config for keeping spec changes consistent

## Get help

The protocol is small but has subtle behavioral requirements. If you're implementing and have a spec question, edge case, or suspect a contradiction:

- **Email a maintainer** — [founder@runcycles.io](mailto:founder@runcycles.io) for protocol design / clarification questions
- **Open a spec issue** — [`runcycles/cycles-protocol/issues`](https://github.com/runcycles/cycles-protocol/issues) for ambiguity, contradiction, or proposed clarifications
- **Open a docs issue** — [`runcycles/cycles-docs/issues`](https://github.com/runcycles/cycles-docs/issues) for documentation gaps
- **Reference implementation issues** — [`runcycles/cycles-server/issues`](https://github.com/runcycles/cycles-server/issues) for behavior that disagrees with the spec

A founder reads every email and issue.

## Why this matters

Protocol > tool.

Every framework, vendor, and platform that implements the Cycles Protocol speaks the same wire format. Client SDKs work against any conformant server. Operators can switch implementations without touching application code. The category — runtime budget authority over AI agents — has a single, open, version-stable contract that transcends any individual product.

OpenTelemetry won observability by being the protocol every vendor implemented. The protocol that becomes the shared implementation contract shapes the category — and it only earns that role when multiple implementations validate the same wire format against the same conformance target, with a shared conformance kit to follow.

If you build a Cycles-compatible server, you make the protocol stronger. That's the work.

## Related

- [Cycles Protocol overview](/protocol/) — the hub page
- [API Reference](/protocol/api-reference-for-the-cycles-protocol) — narrative API docs
- [How Reserve / Commit Works](/protocol/how-reserve-commit-works-in-cycles) — the lifecycle everything else builds on
- [Runtime Authority vs Runtime Authorization](/concepts/runtime-authority-vs-runtime-authorization) — how the protocol fits alongside identity-based agent governance



# Cycles Protocol

The Cycles Protocol is an open specification for **runtime budget authority over AI agents**.

> **Authorization grants access; authority meters and bounds the action.**

The protocol defines how budgets are reserved before execution, committed after, and reconciled across hierarchical scopes — atomically, with three-way decisions and observable events. Cycles is the reference implementation; anyone can implement a conformant server.

## At a glance

- **Open specification** — Apache 2.0, multiple OpenAPI YAMLs in [`runcycles/cycles-protocol`](https://github.com/runcycles/cycles-protocol).
- **Explicit conformance criteria** — current MUST / SHOULD / MAY breakdown lives in [`CONFORMANCE.md`](https://github.com/runcycles/cycles-protocol/blob/main/CONFORMANCE.md). Start there for the authoritative surface against the current conformance target.
- **Reserve-commit lifecycle** — atomic budget locking before action, commit on completion, release on cancel, with TTL heartbeat for long-running operations.
- **Hierarchical scopes** — tenant → workspace → app → workflow → agent → toolset, evaluated atomically in one operation (run-level budgets are modeled by encoding the run id in a Subject field such as `workflow:run-{id}` — `dimensions` are metadata and never derive scopes).
- **Three-way decisions** — `ALLOW` / `ALLOW_WITH_CAPS` / `DENY`. Implementations return constraints (`max_tokens`, `tool_denylist`, `max_steps_remaining`) that let the agent self-regulate, not just stop.
- **Concurrency-safe enforcement** — shared budgets MUST NOT be oversubscribed under concurrent reserve calls.
- **Idempotent commit and release** — retries are safe; the same action MUST NOT settle twice.
- **Explicit error semantics** — `BUDGET_EXCEEDED` (409), `IDEMPOTENCY_MISMATCH` (409), `RESERVATION_EXPIRED` (410), `UNIT_MISMATCH` (400), and the rest defined in the spec.
- **Action-governance preview** — v0.1.26 YAMLs define action kinds, action quotas, allow/deny lists, observe mode, and quota-counter APIs. See [Action Governance Preview](/protocol/action-governance-preview-in-cycles).
- **Multiple language clients** — Python, TypeScript, Rust, Spring Boot, MCP host.

## Specification

The full specification lives in the [`runcycles/cycles-protocol`](https://github.com/runcycles/cycles-protocol) repository. Key files:

| File | Purpose |
|---|---|
| [`cycles-protocol-v0.yaml`](https://github.com/runcycles/cycles-protocol/blob/main/cycles-protocol-v0.yaml) | Runtime base — reserve / commit / release / decide / balances / events |
| [`cycles-governance-admin-v0.1.25.yaml`](https://github.com/runcycles/cycles-protocol/blob/main/cycles-governance-admin-v0.1.25.yaml) | Cross-plane events, webhooks, balances, auth introspection (mixed conformance) |
| [`cycles-action-kinds-v0.1.26.yaml`](https://github.com/runcycles/cycles-protocol/blob/main/cycles-action-kinds-v0.1.26.yaml) | Action-kind registry + quota primitives (upcoming, SHOULD today) |
| [`cycles-protocol-extensions-v0.1.26.yaml`](https://github.com/runcycles/cycles-protocol/blob/main/cycles-protocol-extensions-v0.1.26.yaml) | DenyDetail, ObserveMode, v0.1.26 evaluation order (upcoming) |
| [`cycles-governance-extensions-v0.1.26.yaml`](https://github.com/runcycles/cycles-protocol/blob/main/cycles-governance-extensions-v0.1.26.yaml) | Action-quota / access-control policy fields (upcoming) |
| [`cycles-spec-index.yaml`](https://github.com/runcycles/cycles-protocol/blob/main/cycles-spec-index.yaml) | Index of all spec files with conformance metadata |
| [`CHANGELOG.md`](https://github.com/runcycles/cycles-protocol/blob/main/CHANGELOG.md) | Versioned change history |

For the human-readable v0.1.26 extension walkthrough, see [Action Governance Preview](/protocol/action-governance-preview-in-cycles). The YAML files remain authoritative.

## Conformance

The authoritative statement of what an implementation MUST, SHOULD, and MAY do is [`CONFORMANCE.md`](https://github.com/runcycles/cycles-protocol/blob/main/CONFORMANCE.md) in the spec repo. Read it first — the current conformance target, exact required-operation list, and any version-bumped requirements all live there. At a glance, the surface includes:

- **Core runtime operations** — reserve / commit / release / extend (atomic budget locking and lifecycle)
- **Cross-plane operations** — event listing, webhook delivery / replay, balance queries, auth introspection
- **Recommended operations** — `decide`, `listReservations`, `getReservation`, `createEvent`
- **4 core invariants** — atomic reservation across scopes, concurrency-safe enforcement, idempotent commit/release, unit consistency
- **Exact HTTP status + error code pairs** — implementations MUST return the spec's error codes verbatim so clients can route on them
- **`X-Cycles-API-Key` header** for authentication; key provisioning and scoping is implementation-specific

Reading the conformance doc is the prerequisite for any new implementation. RFC 2119 language throughout.

## Reference implementation

[`runcycles/cycles-server`](https://github.com/runcycles/cycles-server) is the reference implementation — Java / Spring Boot, Apache 2.0, validated against the current conformance target. The companion [`runcycles/cycles-server-admin`](https://github.com/runcycles/cycles-server-admin) provides the management plane (tenant / budget / policy / key CRUD).

Client SDKs that speak the protocol:

- **Python** — [`runcycles` PyPI package](/quickstart/getting-started-with-the-python-client)
- **TypeScript** — [`runcycles` npm package](/quickstart/getting-started-with-the-typescript-client)
- **Spring Boot** — [`cycles-spring-boot-starter`](/quickstart/getting-started-with-the-cycles-spring-boot-starter)
- **Rust** — [Rust client](/quickstart/getting-started-with-the-rust-client)
- **MCP host** — [MCP server](/quickstart/getting-started-with-the-mcp-server) for Claude / Cursor / Windsurf

All clients communicate over the same wire protocol. A conformant alternative server can replace the reference implementation transparently.

## Implement the Cycles Protocol

If you're building a framework, an in-house budget system, or an alternative server that should speak the same wire format as Cycles, see **[Implement the Cycles Protocol](/protocol/implement)** for the minimum implementation surface and conformance walkthrough.

## Protocol reference

The reference pages in the sidebar walk through every primitive in the protocol — reserve-commit lifecycle, scope derivation, units, caps and three-way decisions, overage policies, TTL and grace, decide preflight, dry-run / shadow mode, action-governance preview, events and direct debit, debt and overdraft, balance queries, reservation recovery, metrics and metadata, error codes, webhook event delivery, event payload schemas, scope filter syntax, correlation and tracing, tenant-close cascade semantics.

Start with [API Reference](/protocol/api-reference-for-the-cycles-protocol) for the operation surface, or [How Reserve / Commit Works](/protocol/how-reserve-commit-works-in-cycles) for the lifecycle that everything else builds on.

## Why a protocol, not just a tool

OpenTelemetry didn't win observability by being a tool — it won by being a protocol that every observability vendor implemented. For runtime budget authority on AI agents, the same dynamic applies: the team that owns the protocol owns the category. Cycles ships the spec, the conformance criteria, and the reference implementation in the open. Anyone can implement; everyone speaks the same wire format.

## Related

- [Runtime Authority vs Runtime Authorization](/concepts/runtime-authority-vs-runtime-authorization) — how the protocol fits alongside identity-based agent governance (AWS Bedrock AgentCore Policy, Akeyless)
- [Comparisons](/concepts/comparisons) — how Cycles differs from LiteLLM, Helicone, LangSmith, rate limiters, provider caps, DIY wrappers
- [What Is Runtime Authority for AI Agents](/blog/what-is-runtime-authority-for-ai-agents) — the canonical definition
- [What is Cycles?](/quickstart/what-is-cycles) — 5-minute overview of the reference implementation



# Querying Balances in Cycles: Understanding Budget State

Budget enforcement is only useful if the system can also answer: **how much is left?**

That is what the balances endpoint provides.

`GET /v1/balances` returns the current budget state for one or more scopes — how much is allocated, how much is reserved, how much has been spent, and how much remains.

## What a balance represents

A balance is a snapshot of a single scope's budget state in a specific unit.

Each balance includes:

### remaining (SignedAmount)

The budget available for new reservations.

This is the most important field for operational decisions. It tells you how much room is left.

Remaining can be negative when a scope is in overdraft (debt exceeds available budget).

Formula: `remaining = allocated - spent - reserved - debt`

### reserved (Amount)

The amount currently locked by active reservations.

This budget is held but not yet spent. It will either become spent (via commit) or return to remaining (via release or expiration).

### spent (Amount)

The cumulative amount successfully committed through reservations or events.

This is actual, accounted usage.

### allocated (Amount)

The total budget cap for this scope, if a fixed allocation exists.

When present, allocated represents the maximum budget the scope can consume. When absent, the remaining amount is authoritative and cannot be derived from other fields.

### debt (Amount)

Overdraft consumption that occurred when insufficient budget was available.

Debt is created only when the `ALLOW_WITH_OVERDRAFT` overage policy is used and budget is insufficient at commit time.

When debt is present and no overdraft limit is configured, new reservations are blocked until it is repaid. When an overdraft limit is set, debt within the limit does not block new reservations.

### overdraft_limit (Amount)

The maximum debt this scope is allowed to carry.

If absent or zero, no overdraft is permitted.

### is_over_limit (Boolean)

Whether the scope's debt exceeds its overdraft limit.

When true, all new reservations against this scope are blocked.

## The ledger invariant

When allocated, spent, reserved, and debt are all present, the following invariant holds:

```
remaining = allocated - spent - reserved - debt
```

The server guarantees this relationship. Clients can rely on it for consistency checking.

If allocated is absent, remaining is authoritative — the client must not try to derive it.

## Querying balances

### Request

```
GET /v1/balances?tenant=acme&app=support-bot
```

At least one subject filter must be provided:

- `tenant`
- `workspace`
- `app`
- `workflow`
- `agent`
- `toolset`

Additional parameters:

- `include_children` — if true, include child scopes in the response (default: false; may be ignored by v0 implementations)
- `limit` — maximum results per page (1–200, default: 50)
- `cursor` — opaque cursor from a previous response for pagination

Queries are always scoped to the effective tenant. The server rejects requests that attempt to query another tenant's balances with `403 FORBIDDEN`.

### Response

```json
{
  "balances": [
    {
      "scope": "tenant:acme",
      "scope_path": "tenant:acme",
      "remaining": { "unit": "USD_MICROCENTS", "amount": 85000000 },
      "reserved": { "unit": "USD_MICROCENTS", "amount": 5000000 },
      "spent": { "unit": "USD_MICROCENTS", "amount": 10000000 },
      "allocated": { "unit": "USD_MICROCENTS", "amount": 100000000 }
    },
    {
      "scope": "app:support-bot",
      "scope_path": "tenant:acme/app:support-bot",
      "remaining": { "unit": "USD_MICROCENTS", "amount": 22000000 },
      "reserved": { "unit": "USD_MICROCENTS", "amount": 3000000 },
      "spent": { "unit": "USD_MICROCENTS", "amount": 5000000 },
      "allocated": { "unit": "USD_MICROCENTS", "amount": 30000000 }
    }
  ],
  "has_more": false
}
```

### Pagination

Responses are paginated:

- `limit` — maximum results per page (1–200, default 50)
- `cursor` — opaque cursor from a previous response
- `has_more` — whether more results exist
- `next_cursor` — cursor for the next page

## Use cases

### Operator dashboards

Balances provide the data needed for budget dashboards:

- which tenants are near their limits
- which workflows are consuming the most
- where reserved amounts are high (indicating active work)
- which scopes have debt

### Automated degradation

Systems can query balances to make proactive decisions:

- if remaining is below a threshold, switch to a smaller model
- if reserved is a large fraction of remaining, reduce concurrency
- if debt is present, pause new work

### Budget monitoring and alerting

Balances enable alerting rules:

- warn when remaining drops below 20% of allocated
- alert when debt exceeds 80% of overdraft_limit
- alert when is_over_limit becomes true

### Capacity planning

Historical balance queries (collected over time) reveal consumption patterns:

- average daily spend by tenant
- peak reserved amounts by workflow
- debt frequency by scope

## Understanding scope and scope_path

Each balance has two identifiers:

- **scope** — the individual scope identifier (e.g., `tenant:acme`, `app:support-bot`)
- **scope_path** — the full hierarchical path (e.g., `tenant:acme/app:support-bot`)

The scope_path places the balance in the hierarchy. The scope identifies the individual level.

## Unit consistency

All amount fields within a single balance share the same unit.

A scope may have balances in multiple units (e.g., both USD_MICROCENTS and TOKENS), but each balance object has a single unit.

## Balances are eventually consistent

Balance queries reflect the current server state, which includes all committed and reserved amounts.

However, under high concurrency, balances may be slightly behind the most recent operations. They are suitable for dashboards, monitoring, and planning — not for real-time budget decisions.

For real-time budget decisions, use reservations (which are atomic and concurrency-safe).

## Reading a balance — a practical example

Consider this balance:

```json
{
  "scope": "workflow:refund-assistant",
  "scope_path": "tenant:acme/app:support-bot/workflow:refund-assistant",
  "remaining": { "unit": "USD_MICROCENTS", "amount": 12000000 },
  "reserved": { "unit": "USD_MICROCENTS", "amount": 3000000 },
  "spent": { "unit": "USD_MICROCENTS", "amount": 15000000 },
  "allocated": { "unit": "USD_MICROCENTS", "amount": 30000000 }
}
```

This tells us:

- The workflow was allocated $0.30 (`30,000,000 / 10^8`)
- It has spent $0.15 so far
- $0.03 is currently reserved by active work
- $0.12 remains available for new reservations

Verify the invariant: `30,000,000 - 15,000,000 - 3,000,000 - 0 = 12,000,000` — correct.

## Negative remaining

When a scope has debt, remaining can be negative:

```json
{
  "remaining": { "unit": "USD_MICROCENTS", "amount": -2000000 },
  "debt": { "unit": "USD_MICROCENTS", "amount": 5000000 },
  "overdraft_limit": { "unit": "USD_MICROCENTS", "amount": 10000000 },
  "is_over_limit": false
}
```

This scope is in overdraft but not over-limit. It has $0.05 in debt within a $0.10 overdraft limit. Because the overdraft limit is configured, new reservations are still allowed as long as remaining budget covers the estimate.

## Summary

The balances API gives operators and systems visibility into budget state across all scopes:

- **remaining** — how much room is left
- **reserved** — how much is held by active work
- **spent** — how much has been consumed
- **allocated** — the total budget cap
- **debt** and **overdraft_limit** — overdraft state
- **is_over_limit** — whether the scope is blocked

This data powers dashboards, monitoring, alerting, automated degradation, and capacity planning.

For real-time budget enforcement, use reservations. For understanding budget state, use balances.

## Next steps

To explore the Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)



# Reservation Recovery and Listing in Cycles

In production systems, things go wrong.

A client crashes after creating a reservation but before storing the reservation ID. A network partition delays a commit. An operator needs to find stuck reservations that are holding budget.

The Cycles protocol provides two endpoints for these situations:

- `GET /v1/reservations` — list and filter reservations
- `GET /v1/reservations/{reservation_id}` — get details for a specific reservation

Both are optional in v0 deployments, but they are essential for production operations.

Tenant callers use `X-Cycles-API-Key`. Operators may also use `X-Admin-API-Key` on reservation list/detail; with admin auth, `GET /v1/reservations` requires a `tenant` query parameter because the admin key has no effective tenant.

## Recovering a lost reservation ID

The most common recovery scenario: a client created a reservation, received the response, but crashed before persisting the `reservation_id`.

The reservation exists on the server. Budget is held. But the client has no way to commit or release it.

The solution is to query by idempotency key:

```
GET /v1/reservations?idempotency_key=my-unique-key-123
```

Idempotency keys are expected to be unique per (effective tenant, endpoint, idempotency_key), so the server SHOULD return at most one matching reservation. The spec makes this a SHOULD rather than a MUST, so robust recovery code should tolerate the unlikely case of multiple matches.

The client recovers the `reservation_id` and can then commit or release as needed.

This is why idempotency keys should be generated and persisted before creating the reservation — they serve as the recovery handle.

## Listing reservations

The listing endpoint supports several filters:

### By status

```
GET /v1/reservations?status=ACTIVE
```

Reservation statuses are:

- **ACTIVE** — the reservation is live and budget is held
- **COMMITTED** — actual usage has been recorded
- **RELEASED** — the reservation was canceled and budget returned
- **EXPIRED** — the TTL (plus grace period) elapsed without commit or release

Filtering by `status=ACTIVE` is the most operationally useful — it shows all reservations currently holding budget.

### By subject fields

```
GET /v1/reservations?tenant=acme&app=support-bot
GET /v1/reservations?workflow=refund-assistant
```

Subject filters match against the canonical Subject fields: tenant, workspace, app, workflow, agent, and toolset.

Filtering on custom dimensions is out of scope for v0.

### By idempotency key

```
GET /v1/reservations?idempotency_key=run-abc-step-3
```

The server SHOULD return at most one reservation matching the key. This is the primary recovery mechanism.

### Pagination

Responses are paginated:

- `limit` — maximum results per page (1–200, default 50)
- `cursor` — opaque cursor from a previous response
- `has_more` — whether more results exist
- `next_cursor` — cursor for the next page

### Time-window filters

`GET /v1/reservations` supports three independent ISO 8601 time windows:

- `from` / `to` filter `created_at_ms`.
- `expires_from` / `expires_to` filter `expires_at_ms`.
- `finalized_from` / `finalized_to` filter `finalized_at_ms`.

Each bound is inclusive and may be supplied alone. Blank values are treated as unset. If the lower bound is greater than the upper bound for a pair, the server returns `400 INVALID_REQUEST`. `finalized_at_ms` is only populated on COMMITTED and RELEASED rows, so ACTIVE and EXPIRED rows do not match `finalized_*` filters.

Example: find active reservations that have already passed their expiry:

```bash
curl -G "http://localhost:7878/v1/reservations" \
  -H "X-Cycles-API-Key: $TENANT_API_KEY" \
  --data-urlencode "status=ACTIVE" \
  --data-urlencode "expires_to=2026-07-03T12:00:00Z" \
  --data-urlencode "sort_by=expires_at_ms" \
  --data-urlencode "sort_dir=asc"
```

### Sorting (v0.1.25.12+)

`GET /v1/reservations` accepts two optional query parameters to order results server-side:

- `sort_by` — one of `reservation_id`, `tenant`, `scope_path`, `status`, `reserved`, `created_at_ms`, `expires_at_ms`. When omitted, the server uses its default Redis-SCAN order.
- `sort_dir` — `asc` or `desc`. Defaults to `desc` when `sort_by` is provided.

The `reserved` key sorts by the integer `amount` within each row (well-defined because the v0 single-unit-per-reservation invariant holds). The `scope_path` key sorts lexicographically over the canonical scope path string (e.g. `tenant:acme/workspace:prod/agent:x`). The `tenant` key sorts over `Subject.tenant` only.

Unknown `sort_by` or `sort_dir` values return HTTP 400 `INVALID_REQUEST`. Older servers that don't recognize the params ignore them without error (additive-parameter guarantee).

**Cursor binding.** When `sort_by` is provided, the returned cursor binds to the `(sort_by, sort_dir, filters)` tuple. Reusing a cursor under a different sort key or filter set returns HTTP 400. Reset the cursor whenever you change sort key, direction, or filters.

**Hydration warning.** Current reference servers hydrate all matching rows for sorted reservation listings, then sort and slice. If a sorted query hydrates 2,000 or more rows, the server logs a WARN. Rows beyond 2,000 are not truncated in v0.1.25.39+; narrow filters (`status`, `idempotency_key`, workspace/app/workflow/agent/toolset, or the time-window filters above) still keep incident queries faster and easier to reason about.

```
GET /v1/reservations?status=ACTIVE&sort_by=expires_at_ms&sort_dir=asc&limit=100
```

This returns the 100 oldest-expiring active reservations — useful for incident response when you need to force-release soon-to-expire reservations before they churn.

### Field projection

List rows always include scalar lifecycle fields such as `committed` when present. Larger or optional maps are opt-in through `include`:

- `include=metadata` projects reserve-time metadata.
- `include=committed_metadata` projects commit-time metadata.
- `include=evidence` projects recorded CyclesEvidence refs keyed by `reserve`, `commit`, and `release`.

`include` is projection-only: it does not change which rows match, does not affect ordering, and does not invalidate sorted cursors if it changes between pages.

## Getting reservation details

```
GET /v1/reservations/{reservation_id}
```

Returns the full state of a specific reservation:

- **reservation_id** — the unique identifier
- **status** — current lifecycle state (ACTIVE, COMMITTED, RELEASED, EXPIRED)
- **subject** — the original Subject (tenant, workspace, app, etc.)
- **action** — the original Action (kind, name, tags)
- **reserved** — the amount that was reserved
- **committed** — the amount that was committed (if status is COMMITTED)
- **created_at_ms** — when the reservation was created
- **expires_at_ms** — when the reservation expires (or expired)
- **finalized_at_ms** — when the reservation reached a terminal state; present only on COMMITTED and RELEASED reservations, absent on ACTIVE and EXPIRED ones
- **scope_path** — the canonical scope path
- **affected_scopes** — all scopes impacted by this reservation
- **idempotency_key** — the creation idempotency key (if the server persists it)
- **metadata** — any metadata attached to the reservation
- **committed_metadata** — metadata attached to the commit request, when present
- **evidence** — CyclesEvidence refs keyed by artifact type, when evidence was emitted

This endpoint is useful for debugging specific reservations and understanding their full lifecycle.

## Tenancy enforcement

Both listing and detail endpoints enforce tenant isolation:

- results are scoped to the effective tenant derived from the API key
- if a `tenant` query parameter is provided, it must match the effective tenant
- attempting to get details for a reservation owned by a different tenant returns `403 FORBIDDEN`

A tenant cannot see or access another tenant's reservations.

Under `X-Admin-API-Key`, reservation list requires `tenant` as an explicit filter and reservation detail can read any reservation by ID. This is the admin-on-behalf-of operator path used by the dashboard and incident runbooks.

## Use cases

### Stuck reservation detection

Periodically query `GET /v1/reservations?status=ACTIVE` and check for reservations with `expires_at_ms` in the past that have not yet been finalized.

These may indicate server-side cleanup delays or edge cases worth investigating.

### Budget leak investigation

When budget appears lower than expected, list active reservations to see what is currently held:

```
GET /v1/reservations?status=ACTIVE&tenant=acme&app=support-bot
```

This shows all live reservations holding budget for that app. High counts or large reserved amounts may explain the discrepancy.

### Post-incident analysis

After a budget incident, query reservations by workflow or agent to understand the pattern:

```
GET /v1/reservations?workflow=refund-assistant&status=COMMITTED
```

This shows all committed reservations for that workflow, which helps trace what consumed the budget.

### Client crash recovery

Current official SDK lifecycle helpers replay known-actual settlement from their durable journal automatically. The manual sequence below applies to raw clients and application-owned lifecycles:

1. Check durable local state for in-progress reservation IDs, exact settlement bodies, and idempotency keys
2. For any missing reservation IDs, query by the idempotency key that was generated before the reservation
3. For each recovered reservation, check its status:
   - **ACTIVE**: commit or release depending on whether work completed
   - **EXPIRED**: if work already completed and actual usage is known, record it through `POST /v1/events` with the stored settlement key; if work did not run, no settlement is due; create a new reservation only before starting new work
   - **COMMITTED** or **RELEASED**: no action needed

This recovery pattern depends on persisting idempotency keys before creating reservations and persisting known-actual settlement before its first request. Never compensate with a fresh key while an original request remains ambiguous.

## Error conditions

### GET /v1/reservations (listing)

- `400 INVALID_REQUEST` — malformed query parameters
- `401 UNAUTHORIZED` — invalid API key
- `403 FORBIDDEN` — tenant mismatch

### GET /v1/reservations/{reservation_id} (detail)

- `401 UNAUTHORIZED` — invalid API key
- `403 FORBIDDEN` — reservation owned by a different tenant
- `404 NOT_FOUND` — reservation never existed
- `410 RESERVATION_EXPIRED` — reservation has expired

## Summary

The reservation listing and detail endpoints provide operational visibility and recovery capabilities:

- **Recovery**: query by idempotency key to recover lost reservation IDs
- **Monitoring**: list active reservations to understand what is holding budget
- **Debugging**: get full reservation details including status, amounts, scopes, and timestamps
- **Investigation**: filter by subject fields and status for post-incident analysis

These endpoints are optional in v0, but essential for production operations where client crashes, network issues, and budget investigations are a reality.

## Next steps

To explore the Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)
- Review [SDK Settlement Recovery and Durability](/protocol/sdk-settlement-recovery-and-durability)



# Reservation TTL, Grace Period, and Extend in Cycles

Reservations in Cycles do not live forever.

Every reservation has a time-to-live (TTL). When the TTL (plus grace period) elapses without a commit or release, the reservation expires and the held budget returns to the available pool.

This is by design.

Without TTL, a crashed client or lost network connection could lock budget indefinitely. That would create phantom consumption — budget that appears used but is not serving any real work.

TTL prevents that. But it also creates a design question: how should long-running work keep its reservation alive?

That is where extend comes in.

## How TTL works

When a reservation is created, the server sets an expiration time:

```
expires_at_ms = created_at_ms + ttl_ms
```

The default TTL is 60 seconds (`ttl_ms: 60000`). The allowed range is 1 second to 24 hours (`1000` to `86400000` milliseconds).

Tenant-level TTL controls are reference-server behavior, not part of the v0 protocol: through the Admin API, tenant administrators can change the default with `default_reservation_ttl_ms` and restrict the range with `max_reservation_ttl_ms` — any requested TTL exceeding the tenant maximum is capped automatically.

When server time passes `expires_at_ms`, the reservation has expired: it can no longer be extended. In-flight commits and releases are still accepted during the grace period (below). Once the grace period has also elapsed without a commit or release, the server marks the reservation `EXPIRED` and returns the reserved budget to the available pool.

## How grace period works

Real systems have in-flight operations. A commit request may be in transit when the TTL expires.

The grace period provides a short window after TTL expiration during which commits and releases are still accepted:

```
hard_expiry = expires_at_ms + grace_period_ms
```

The default grace period is 5 seconds (`grace_period_ms: 5000`).

The allowed range is 0 to 60 seconds (`0` to `60000` milliseconds).

During the grace period:

- commit and release are still accepted
- extend is not accepted (the reservation must be extended before TTL expires)

After the grace period, the reservation is marked `EXPIRED`. Any attempt to commit or release returns `410 RESERVATION_EXPIRED`. (Expiry is not a finalization — only `COMMITTED` and `RELEASED` are finalized states, which is why an expired reservation returns `410 RESERVATION_EXPIRED` rather than `409 RESERVATION_FINALIZED`.)

### Why the grace period maximum is 60 seconds

The 60-second ceiling is an interoperability constraint. Longer grace periods would extend the window during which a crashed client can keep budget locked, increasing the risk of zombie reservations.

For operations that need more time, use extend as a heartbeat rather than relying on a long grace period.

## How extend works

`POST /v1/reservations/{reservation_id}/extend` extends the TTL of an active reservation.

The `extend_by_ms` parameter is added to the current `expires_at_ms` (not to the request time):

```
new_expires_at_ms = current_expires_at_ms + extend_by_ms
```

The allowed range for `extend_by_ms` is 1 millisecond to 24 hours (`1` to `86400000` milliseconds).

Extend requires an `idempotency_key`. Replaying the same request with the same idempotency key returns the original response.

### What extend does not change

Extend updates only the expiration time. It does not change:

- the reserved amount
- the unit
- the subject
- the action
- the scope path
- the affected scopes

The reservation is the same reservation. It just lives longer.

### Error conditions

- If the reservation is already `COMMITTED` or `RELEASED`: `409 RESERVATION_FINALIZED`
- If the reservation has already expired (past `expires_at_ms`): `410 RESERVATION_EXPIRED`
- If the tenant's `max_reservation_extensions` limit has been reached: `409 MAX_EXTENSIONS_EXCEEDED`
- If the reservation was never created: `404 NOT_FOUND`

Note: extend must happen before TTL expires, not during the grace period. The grace period only covers commit and release.

### Extension limits

Tenant administrators can set `max_reservation_extensions` via the Admin API to limit how many times a single reservation can be extended (default: 10). This prevents infinite zombie reservations from clients that heartbeat indefinitely without committing or releasing.

## The heartbeat pattern

For long-running workflows:

1. Create a reservation with a bounded TTL
2. Schedule heartbeat extensions from the server's returned remaining lifetime
3. When work completes, commit actual usage
4. If the heartbeat stops (crash, timeout), the reservation expires naturally and budget is returned

This pattern keeps budget locked only while the client is actively running. If the client crashes, the reservation expires quickly and budget is freed.

### Server-authoritative scheduling

Since runtime spec revision 0.1.25.16, successful create and extend responses can carry `remaining_ttl_ms`. It is computed from the authoritative server clock and is the normative scheduling input whenever present.

Recompute the schedule from every exact, schema-valid HTTP `200` create or extend response:

```text
rtt = monotonic receive time - monotonic send time
lead_floor = max(0, remaining_ttl_ms - rtt)
attempt_budget = max(enforced request timeout, 1000ms, 2 × max observed rtt)
safety_margin = max(1000ms, 2 × max observed rtt)
retry_reserve = 2 × attempt_budget + safety_margin
next_delay = max(0, lead_floor - retry_reserve)
```

The reserve leaves room for one failed attempt, a same-key recovery attempt, and scheduling/network margin. For example, a returned 60-second lead, 10-second request timeout, and 1.5-second maximum observed RTT produce a 23-second retry reserve and a cadence of about 37 seconds.

::: warning Keep extend timeouts well below the lease
The default 60-second TTL cannot support a safe positive delay if one extend attempt may consume 30 seconds: the retry reserve is at least 61 seconds. Configure the enforced per-extend timeout well below half the smallest expected lease. Current official SDK defaults use finite 7–12 second attempt budgets.
:::

After a timeout, connection failure, 5xx, 429, or malformed/other 2xx response, the client keeps the same idempotency key and recomputes the retry window from the last valid server response:

```text
current_lead = max(0, last lead_floor - monotonic elapsed time)
retry_window = current_lead - attempt_budget - safety_margin
```

For a timeout, connection error, 5xx, or ambiguous 2xx, the recovery delay is:

```text
retry_delay = min(30000ms, current_lead / 4, retry_window)
```

The client may keep retrying while the freshly recomputed window is positive. A zero window permits one immediate same-key recovery attempt; if that also fails before an intervening success, the client stops. A negative window proves that no complete attempt plus margin fits. The client also stops if neither monotonic elapsed time nor the window decreases between consecutive failures, which prevents a coarse-clock zero-time loop.

A 429 delay is honored only when a valid non-negative delta-seconds `Retry-After` fits inside the window; the heartbeat does not accept the HTTP-date form or invent an earlier retry. Any other 4xx stops and surfaces without rotating the key.

If the initial create is ambiguous, no valid lead exists yet. The client makes at most one immediate retry with the same create key, then stops and surfaces if that retry is also ambiguous.

If a valid response produces zero delay, one immediate extension with a fresh key is allowed. A second consecutive zero-delay success stops the heartbeat instead of burning the server's extension limit in a tight loop. Missing or unreliable monotonic timing also produces zero delay and therefore reaches this guard.

### Same-key replay and `remaining_ttl_ms`

A successful create or extend replay returns the original response except for `remaining_ttl_ms`, which the server recomputes when constructing the replay. The value is `0` when the reservation is no longer active and may conservatively understate lead if a later, separately keyed extension moved expiry farther out.

This field is volatile transport metadata. It is deliberately excluded from the CyclesEvidence payload, so recomputing it does not change the original evidence identity.

### Older servers without the field

When `remaining_ttl_ms` is absent, a client cannot reliably infer the true lease from `expires_at_ms`: server policy may cap either each extension grant or the maximum lead beyond server time. The official SDKs retain a bounded measured-grant heuristic as a non-normative, best-effort fallback. The exact schema-valid HTTP `200` success rule still applies; only scheduling changes.

The fallback intentionally prefers over-beating and possible extension-budget exhaustion over an unobservable lease lapse. This differs from the field-bearing path, which has enough information to prove when no safe retry fits and stop. Do not use the fallback merely because local monotonic timing is unavailable; a field-bearing response still requires the primary algorithm.

## The clients handle this automatically

Python `@cycles`, TypeScript `withCycles`, Spring `@Cycles`, and Rust `ReservationGuard` schedule extensions in the background and stop the heartbeat when the lifecycle closes.

Heartbeat failures use a warning policy: each failure reports the reservation ID and retry/stop disposition, but it does not cancel the guarded work or suppress final settlement. Recoverable failures retry with the same key. Permanent conditions — `RESERVATION_EXPIRED`, `RESERVATION_FINALIZED`, `MAX_EXTENSIONS_EXCEEDED`, `TENANT_CLOSED`, and `NOT_FOUND` — stop the heartbeat.

If known spend reaches commit after the reservation expires, current SDK lifecycle helpers recover it through a durable `POST /v1/events` fallback. See [SDK Settlement Recovery and Durability](/protocol/sdk-settlement-recovery-and-durability).

## Choosing TTL values

### Short TTL (10–30 seconds)

Best for:

- model calls and tool invocations that complete quickly
- actions with predictable duration
- systems where fast budget recovery matters

### Medium TTL (30–120 seconds)

Best for:

- multi-step workflows
- actions with moderate latency
- systems using heartbeat extension

### Long TTL (2–60 minutes)

Best for:

- batch processing
- background jobs with known duration
- systems where extend is not practical

Use long TTLs sparingly. They increase the risk of zombie reservations.

## Choosing grace period values

### Default (5 seconds)

Sufficient for most synchronous operations where commits arrive shortly after execution.

### Higher (10–30 seconds)

Useful for:

- streaming model calls with high latency
- slow external APIs
- actions where commit may be delayed by processing

### Zero

Use zero grace period when:

- strict TTL enforcement is required
- the client always commits well before expiration
- zombie prevention is a priority

## Common mistakes

### Mistake 1: Using large TTLs instead of heartbeats

A 10-minute TTL works but locks budget for the full duration if the client crashes. Prefer short TTLs with extend.

### Mistake 2: Relying on grace period for normal operation

The grace period is a safety net, not a design tool. If commits routinely arrive during the grace period, the TTL is too short.

### Mistake 3: Forgetting to handle RESERVATION_EXPIRED

If a commit arrives after the grace period, it is rejected. Do not create a new reservation after the work already ran; that would treat completed work as new authorization. Recover known spend as an idempotent direct event. Current official SDK lifecycle helpers do this automatically and persist the recovery across restart.

### Mistake 4: Extending after TTL expires

Extend must happen before `expires_at_ms`. If the client waits too long between heartbeats, the reservation may expire before the next extend arrives.

## Summary

Reservations in Cycles are time-bounded by design:

- **TTL** controls how long budget is held
- **Grace period** provides a short safety window after TTL for in-flight commits
- **Extend** refreshes TTL as a heartbeat for long-running operations

The recommended pattern for most systems:

- Keep TTL bounded and choose an enforced request timeout that leaves room for the retry reserve
- Schedule from `remaining_ttl_ms` whenever the server supplies it
- Reuse the same idempotency key for ambiguous extend recovery
- Set grace period to 5–10 seconds
- Recover known spend through an idempotent event if commit reaches an expired reservation

This keeps budget locked only while work is actively running, and recovers quickly when clients crash.

## Next steps

To explore the Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)



# SDK Recovery Conformance Matrix

Wire-format compatibility is not enough for an SDK. The failure choreography
must still preserve known spend when a response is lost, a reservation expires,
credentials rotate, or the process restarts.

This matrix publishes the current evidence snapshot for every official core
SDK against the shared [recovery profile
0.3](https://github.com/runcycles/cycles-protocol/blob/main/client-recovery/PROFILE.md).
Each cell links to the machine-readable report containing the exact native
tests executed for that scenario. The four implementations are Python,
TypeScript, Spring / Java, and Rust.

<RecoveryConformanceMatrix />

## What a pass proves

The shared runner invokes the SDK adapter once per scenario in a fresh process.
The adapter must execute named native behavior tests and return their exact
identifiers. A pass therefore proves that the SDK's claimed native tests ran
successfully at the pinned implementation commit.

The runner deliberately does not disclose the expected request choreography to
the adapter. Code review still verifies that each named native test asserts the
catalog's expected calls and outcomes. This is native-test-backed conformance,
not a claim that a second black-box client independently reproduced the SDK
internals.

## Status definitions

- **Pass** — the pinned report passed the scenario and names at least one native
  behavior test.
- **Fail** — the runner completed but the scenario or its native tests failed.
- **Not claimed** — an implementation does not claim that profile level.
- **Stale** — the displayed SDK or profile commit has moved beyond the report.
  The pinned result remains historical evidence, but current conformance must
  be re-established.

The SDK README badge follows the current `main` CI workflow. The snapshot above
is intentionally pinned to exact commits so later source changes cannot rewrite
what was tested.

## Guarantee boundary

Durable recovery begins only after the lifecycle helper knows the actual amount
and persists settlement. If a process dies before the provider returns actual
usage, the SDK cannot invent the missing amount. Applications requiring that
stronger guarantee must durably checkpoint the provider receipt or usage before
acknowledging the downstream operation.

See [SDK Settlement Recovery and
Durability](/protocol/sdk-settlement-recovery-and-durability) for the journal,
replay, expiry fallback, configuration, and operational contract.



# SDK Settlement Recovery and Durability

A reservation protects budget before work starts. Settlement records what the work actually consumed after it runs. Once actual usage is known, losing that settlement would undercount spend even though the provider or tool already performed the work.

The official Python 0.5.2+, TypeScript 0.4.2+, Java/Spring 0.3.2+, and Rust 0.3.2+ SDK lifecycle helpers implement the Cycles [durable recovery profile](https://github.com/runcycles/cycles-protocol/blob/main/client-recovery/PROFILE.md). They persist unresolved known-actual settlement before the first request and replay it with the original idempotency key until the outcome is proven.

## Where the guarantee starts

The durability guarantee starts only after the lifecycle helper knows the actual amount and begins commit or direct-event settlement.

If the process dies before the provider returns usage, the SDK does not know what amount to settle. Applications that need convergence across that boundary must durably checkpoint the provider receipt or actual usage before acknowledging the downstream operation.

::: warning Lifecycle helpers and low-level clients differ
The automatic guarantee belongs to helpers that own the reserve → execute → settle flow: Python `@cycles` and streaming contexts, TypeScript `withCycles` and stream handles, Spring `@Cycles`, and Rust `ReservationGuard`.

Low-level `commit` and `createEvent` methods expose the idempotent protocol primitives but do not persist application-owned requests automatically. Direct callers must supply equivalent durable retry storage.
:::

## Recovery choreography

Once actual usage is known, a conforming lifecycle helper follows this sequence:

1. Build the commit request and its same-key `/v1/events` fallback.
2. Atomically persist the unresolved settlement before the first request leaves the process.
3. Send the commit.
4. Remove the record only after a proven settlement success or a genuine, understood terminal rejection.
5. Retain and replay the record after an ambiguous outcome, authentication failure, unclassifiable 4xx response, retry exhaustion, or restart.

Only an exact, schema-valid HTTP `200` commit response proves commit success. Only an exact, schema-valid HTTP `201` event response proves fallback success. A malformed body or another 2xx status remains ambiguous and is retried with the original idempotency key.

The SDK never releases a reservation merely because settlement is ambiguous or credentials failed after the action ran. Releasing at that point would return budget for spend that may already have occurred.

## Expired commits become direct events

If commit returns HTTP `410` or `RESERVATION_EXPIRED`, the reservation hold has already been reclaimed. Retrying that reservation cannot record the spend.

The lifecycle helper therefore switches the durable record to event mode before calling `POST /v1/events`. The event preserves the original subject, action, actual amount, metrics, metadata, and idempotency key, and adds recovery metadata such as `recovered_reservation_id`.

Commit and direct-event idempotency are scoped to their respective endpoints. Reusing the stored key makes repeated recovery safe, including concurrent replay by multiple processes.

## Journal behavior

Durable recovery is enabled by default and uses `~/.runcycles/commit-journal` unless configured otherwise.

- Records are written atomically and use `v2-<sha256-of-exact-UTF-8-reservation-id>.json` filenames.
- Malformed or unsupported-version records are quarantined without preventing other valid records from replaying.
- A persisted HTTP 429 `Retry-After` becomes an absolute not-before time, so restart does not retry early.
- Records are partitioned by server and principal. Configure the tenant identity so the partition remains stable across API-key rotation.
- API keys are not stored in journal records.
- Journal files contain settlement data, including subject, action, actual usage, and metadata. Protect the directory as sensitive application state and avoid putting secrets in metadata.

If journal I/O fails, the SDK reports the durability failure and may still attempt synchronous settlement. It cannot promise restart recovery for a record that was never written.

## Configuration and shutdown

| SDK | Journal settings | Bounded drain |
|---|---|---|
| Python | `journal_enabled`, `journal_dir`, `retry_flush_timeout` | Automatic process-exit flush; unresolved records remain for next startup |
| TypeScript | `journalEnabled`, `journalDir`, `retryFlushTimeout` | `flushPendingCommits(timeoutMs?)` |
| Java/Spring | `cycles.journal.enabled`, `cycles.journal.dir`, `cycles.retry.flush-timeout` | Spring context shutdown |
| Rust | `journal_enabled`, `journal_dir` | `flush_pending_commits_with_timeout(...)`; also available on the blocking client |

Disabling inline/background retry does not disable persistence. To opt out of recovery durability, the journal must also be disabled. Do that only when the application supplies an equivalent durable settlement mechanism.

## Availability behavior

Official lifecycle helpers fail closed before execution. If reservation
creation cannot produce a schema-valid `ALLOW` or `ALLOW_WITH_CAPS` response
after its bounded same-key recovery, the helper surfaces an error and does not
invoke the guarded function. An explicit denial, authentication failure, or
malformed response is never converted into permission to run.

This is a phase-specific policy rather than one blanket outage switch:

- **Before authorization:** no proven allow means no guarded action.
- **After authorization:** heartbeat failure is observable but does not cancel
  work under the baseline warning policy.
- **After actual usage is known:** settlement is durably retained and replayed
  instead of being discarded during an outage.

Low-level clients return the protocol primitives and leave application control
flow to the caller. The official lifecycle helpers do not expose a generic
`fail_open` option. Applications that deliberately execute without a
reservation are outside the pre-execution authority guarantee and must make
that unmetered path explicit in their own policy and telemetry.

## Heartbeat failures do not erase settlement

Heartbeat extension protects the lease while work runs; it is not settlement. Under the official SDKs' warning policy:

- transport and terminal extend failures are logged with the reservation ID and whether the heartbeat will retry or stop;
- recoverable failures reuse the same idempotency key and follow the protocol's safe retry window;
- the guarded action continues; and
- final settlement is still attempted when actual usage becomes known.

A stopped heartbeat can cause commit to arrive after expiry. The durable event fallback above is what preserves that known spend.

## Operational checklist

- Keep journaling enabled on every process that uses high-level lifecycle helpers.
- Configure `tenant` when possible so pending records survive API-key rotation.
- Put the journal on persistent local storage, not an ephemeral container layer.
- Invoke the bounded drain during graceful shutdown where the SDK exposes one.
- Alert on journal I/O failures, quarantined records, retained authentication failures, retry exhaustion, and heartbeat stop messages.
- Do not delete pending records manually unless the settlement outcome has been independently proven.

## Next steps

- [SDK Recovery Conformance Matrix](/protocol/sdk-recovery-conformance)
- [How Reserve → Commit Works](/protocol/how-reserve-commit-works-in-cycles)
- [Reservation TTL, Grace Period, and Extend](/protocol/reservation-ttl-grace-period-and-extend-in-cycles)
- [Error Handling Patterns](/how-to/error-handling-patterns-in-cycles-client-code)
- [SDK Recovery Conformance Profile](https://github.com/runcycles/cycles-protocol/blob/main/client-recovery/PROFILE.md)



# Standard Metrics and Metadata in Cycles

Budget enforcement tells you whether work is allowed and how much it costs.

But production systems need more than cost numbers. They need to know what happened during execution — how many tokens were consumed, how long it took, which model version was used, and any custom data relevant for debugging or analytics.

That is what standard metrics and metadata provide.

## Where metrics and metadata appear

Metrics and metadata can be attached to two operations:

- **Commits** (`POST /v1/reservations/{id}/commit`) — when finalizing a reservation
- **Events** (`POST /v1/events`) — when recording direct debit usage

Both accept an optional `metrics` field and an optional `metadata` field.

Neither field is echoed back in the response. Commit responses carry `status`, `charged`, `released`, `balances`, and `cycles_evidence`; event responses carry `status`, `event_id`, `charged`, and `balances`. To read metadata back after the fact, retrieve the reservation (see below).

## Standard metrics

The protocol defines a `StandardMetrics` schema with four named fields and an extensible custom map:

### tokens_input

```json
"tokens_input": 1250
```

The number of input tokens consumed by the operation. Integer, minimum 0.

Useful for tracking prompt size and correlating with model pricing.

### tokens_output

```json
"tokens_output": 430
```

The number of output tokens generated. Integer, minimum 0.

Useful for tracking generation length and correlating with output pricing (which is typically higher than input pricing).

### latency_ms

```json
"latency_ms": 1840
```

The total operation latency in milliseconds. Integer, minimum 0.

Useful for SLA monitoring, performance analysis, and identifying slow operations that may need different TTL or timeout handling.

### model_version

```json
"model_version": "gpt-4o-mini-2024-07-18"
```

The actual model or tool version used. String, maximum 128 characters.

This is important because the model requested and the model used are not always the same. Providers may route to different versions, and this field captures what actually ran.

### custom

```json
"custom": {
  "cache_hit": "true",
  "region": "us-east-1",
  "retry_count": "2"
}
```

An open map for arbitrary additional metrics. Values can be any JSON type (strings, numbers, booleans, objects).

Use custom metrics for anything not covered by the standard fields — cache behavior, retry counts, routing decisions, feature flags, or domain-specific measurements.

## A complete metrics example

```json
{
  "idempotency_key": "commit-run-42-step-7",
  "actual": { "unit": "USD_MICROCENTS", "amount": 285000 },
  "metrics": {
    "tokens_input": 1250,
    "tokens_output": 430,
    "latency_ms": 1840,
    "model_version": "gpt-4o-mini-2024-07-18",
    "custom": {
      "cache_hit": "false",
      "prompt_template": "summarize-v3"
    }
  }
}
```

## Metadata

Metadata is a separate field from metrics. It is an open map for arbitrary key-value pairs:

```json
{
  "idempotency_key": "commit-run-42-step-7",
  "actual": { "unit": "USD_MICROCENTS", "amount": 285000 },
  "metadata": {
    "user_id": "user-456",
    "session_id": "session-001",
    "feature_flag_bucket": "variant-b",
    "external_trace_id": "otel-abc-xyz-789"
  }
}
```

Metadata is intended for application-level audit, debugging, and correlation to systems outside Cycles — not for operational metrics, and not for the server-managed `trace_id` and `request_id` (those flow through first-class response headers and response-body fields; see the next section).

::: tip Don't put the server trace_id in metadata
As of v0.1.25 the server manages its own 32-hex W3C Trace Context `trace_id` and per-request `request_id`. Both flow on response headers (`X-Cycles-Trace-Id`) and in response bodies — you don't need to (and shouldn't) shove them into `metadata`. Use `metadata` for application-level values like your own OpenTelemetry / Datadog trace id, user id, or session id. See [Correlation and Tracing](/protocol/correlation-and-tracing-in-cycles).
:::

### Metrics vs metadata vs server correlation identifiers

- **Metrics** are about what happened during execution (tokens, latency, model version).
- **Metadata** is about application-level context (user IDs, session IDs, feature flags, external trace ids).
- **Server correlation identifiers** — `request_id`, `trace_id`, `correlation_id` — are managed natively by the Cycles server. You do not need to pack them into `metadata`.

All three are optional. All three are stored with the commit or event record. But they serve different analytical purposes. See [Correlation and Tracing](/protocol/correlation-and-tracing-in-cycles) for the server's three-tier identifier model.

## Where metadata also appears

Metadata is accepted on several other operations beyond commits and events:

- **Reservation creation** (`POST /v1/reservations`) — attach context to the reservation itself
- **Reservation extend** (`POST /v1/reservations/{id}/extend`) — attach debugging metadata to extend operations
- **Decide** (`POST /v1/decide`) — attach context to preflight budget checks

This means a full reservation lifecycle can carry metadata from creation through commit:

1. Create reservation with `metadata: { "external_trace_id": "..." }`
2. Extend with `metadata: { "heartbeat_seq": "3" }`
3. Commit with `metadata: { "app_request_id": "..." }` and `metrics: { ... }`

Commit metadata is preserved on the reservation and returned by `GET /v1/reservations/{id}` as `committed_metadata` — distinct from the reserve-time `metadata` field, which is returned on the same response — so the metadata attached at reserve and commit time is auditable after the fact, not just sent and forgotten.

## Metrics in client code

Attach metrics and metadata through the context object inside a decorated function or annotated method. The SDK automatically includes these in the commit request when the function returns.

::: code-group
```python [Python]
from runcycles import cycles, get_cycles_context, CyclesMetrics

@cycles(estimate=1000)
def chat(prompt: str) -> str:
    response = call_llm(prompt)

    ctx = get_cycles_context()
    ctx.metrics = CyclesMetrics(
        tokens_input=response.usage.prompt_tokens,
        tokens_output=response.usage.completion_tokens,
        latency_ms=elapsed,
        model_version=response.model,
    )
    ctx.commit_metadata = {
        "app_request_id": app_request_id,
        "external_trace_id": otel_trace_id,
    }

    return response.text
```
```java [Java (Spring Boot)]
@Cycles("1000")
public ChatResponse chat(String prompt) {
    ChatResponse response = chatModel.call(prompt);

    CyclesReservationContext ctx = CyclesContextHolder.get();

    CyclesMetrics metrics = new CyclesMetrics();
    metrics.setTokensInput(response.getUsage().getPromptTokens());
    metrics.setTokensOutput(response.getUsage().getCompletionTokens());
    metrics.setLatencyMs(elapsed);
    metrics.setModelVersion(response.getMetadata().getModel());
    ctx.setMetrics(metrics);

    ctx.setCommitMetadata(Map.of(
        "app_request_id", appRequestId,
        "external_trace_id", otelTraceId
    ));

    return response;
}
```
:::

## Why standard metrics matter

### Cost attribution

Tokens input and output, combined with model version, enable precise cost attribution:

- which model was used
- how many tokens it consumed
- what the actual cost was

This connects budget accounting to provider-level billing.

### Performance monitoring

Latency metrics across commits reveal:

- which actions are slow
- whether latency correlates with budget consumption
- where timeout or TTL adjustments are needed

### Audit trail

Metadata creates a traceable path from budget operations back to the originating request, user, or workflow run.

When investigating a budget incident, metadata helps answer: who triggered this, from which session, as part of which trace?

### Analytics

Over time, standard metrics enable aggregate analysis:

- average tokens per model call by action type
- latency distributions by model version
- cache hit rates across workflows
- cost efficiency trends

Note that the protocol defines how metrics are *submitted*, not how they are retrieved or aggregated. v0 defines no endpoint that returns stored metrics, so retrieval and aggregation are implementation-defined — via server-side export, a log pipeline, or direct store access.

## Best practices

### Always include tokens and model version on LLM calls

These are the minimum metrics that make budget data actionable. Without them, cost numbers exist without context.

### Use metadata for correlation IDs

Attach your own correlation keys — e.g. `app_request_id`, `external_trace_id`, or `session_id` — to every commit. This makes it possible to join budget data with application logs and distributed traces. (Use distinct names rather than `request_id`/`trace_id`, which are server-managed and should not be duplicated into metadata — see the trace-context note above.)

### Keep custom metrics stable

Treat custom metric keys like a schema. Changing keys breaks downstream analytics. Add new keys freely, but avoid renaming or removing existing ones without coordination.

### Do not put sensitive data in metrics or metadata

Metrics and metadata are stored and may be visible through admin interfaces or log aggregation. Do not include PII, secrets, or authentication tokens.

## Summary

Standard metrics and metadata enrich budget operations with execution context:

- **tokens_input** and **tokens_output** — token consumption
- **latency_ms** — operation duration
- **model_version** — actual model used
- **custom** — extensible metrics map
- **metadata** — application correlation keys, audit context, and debugging data

These fields are optional but recommended. They turn budget accounting from raw cost numbers into actionable operational data.

## Server-side operational metrics

The metrics above describe the **execution-context fields** a client attaches to each commit or event. They are stored with the protocol record and surface in admin audit trails — they are not echoed on commit or event responses.

Cycles also exposes **Prometheus metrics** on each service's `/actuator/prometheus` endpoint for operational monitoring. These are aggregate counters and histograms — they do not replace per-request metrics, they complement them.

The runtime server (`cycles-server` v0.1.25.10+) currently publishes 11 domain counters and one maintenance timer. The core lifecycle counters are:

- `cycles_reservations_reserve_total{tenant, decision, reason, overage_policy}`
- `cycles_reservations_commit_total{tenant, decision, reason, overage_policy}`
- `cycles_reservations_release_total{tenant, actor_type, decision, reason}`
- `cycles_reservations_extend_total{tenant, decision, reason}`
- `cycles_reservations_expired_total{tenant}`
- `cycles_reservations_quarantined_total{tenant, reason}`
- `cycles_reservations_created_at_index_reads_total{outcome}`
- `cycles_events_total{tenant, decision, reason, overage_policy}`
- `cycles_overdraft_incurred_total{tenant}`

Maintenance adds `cycles_maintenance_runs_total{job, outcome}` and `cycles_maintenance_duration_seconds{job, outcome}`; evidence enqueue failures use `cycles_evidence_emit_failed_total{artifact_type}`.

The admin server currently exposes seven custom counters, including `cycles_admin_audit_writes_total{path_class, outcome}`—**alert on `outcome=error` nonzero** to catch silent audit-coverage loss—and tenant-close reconciliation/outbox signals.

The events service (`cycles-server-events` v0.1.25.6+) currently publishes 17 webhook-delivery, evidence-worker, dispatcher, and security counters plus one delivery-latency timer. See the [Prometheus Metrics Reference](/how-to/prometheus-metrics-reference) for the authoritative inventory and labels.

The runtime and events services gate their optional `tenant` label with `cycles.metrics.tenant-tag.enabled`. The runtime defaults it to `true`; the events service defaults it to `false`. Admin `cycles_admin_*` counters do not carry a tenant label, so this toggle does not apply there.

## Next steps

To explore the Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)



# Tenant-Close Cascade Semantics

Closing a tenant is more than a status flip. Every object the tenant owns — budgets, reservations, API keys, webhook subscriptions — has to move to a terminal state too, and every subsequent mutation against those objects has to be rejected cleanly. The `cycles-governance-admin-v0.1.25.yaml` spec's `CASCADE SEMANTICS` section is the normative contract for how this works.

This page is the operator-facing reference. For the admin API surface that honors the contract, see the [Admin API Guide](/admin-api/guide). For the error-code side, see [Error Codes and Error Handling](/protocol/error-codes-and-error-handling-in-cycles#tenant-closed-409).

## Why this exists

Before the cascade contract was formalized (spec document revision 0.1.25.31), closing a tenant was a pure status flip. Operators would then have to separately:

- drain open reservations (or let TTL expire)
- freeze or close each owned budget
- revoke every API key
- disable every webhook subscription

In practice nobody did all of that. The `/admin/overview` dashboard would accumulate "FROZEN budgets on CLOSED tenants" rows forever — inflating the "needs attention" counter with rows operators had no user-reachable path to resolve.

The cascade contract, formalized at spec document revision 0.1.25.31 and shipping in `cycles-server-admin` v0.1.25.35+, makes the close operation do the right thing atomically (or eventually-atomically) instead.

## Version gate matrix

| Feature | Minimum component | What works |
|---|---|---|
| Rule 1 cascade (budgets + reservations) | `cycles-server-admin` v0.1.25.35 | Closing a tenant cascades budgets → CLOSED and open reservations → RELEASED |
| Rule 2 guard (budget operations, webhook create/update) | `cycles-server-admin` v0.1.25.35 | Admin-plane mutations against closed-tenant budgets, and webhook create/update, return `409 TENANT_CLOSED` |
| Rule 2 full coverage (policies, api-keys, remaining webhook mutations) | `cycles-server-admin` v0.1.25.36 | All remaining admin-plane mutation endpoints also return `409 TENANT_CLOSED` |
| Rule 2 runtime guard (reservation create/commit/release/extend) | `cycles-server` v0.1.25.47 (runtime spec v0.1.25.13) | Persisting reservation mutations on a closed tenant return `409 TENANT_CLOSED`; fresh dry-run/decide evaluations return `200 decision=DENY reason_code=TENANT_CLOSED` |
| Dashboard tombstone + cascade preview UI | `cycles-dashboard` v0.1.25.43 | Banner, CLOSE dialog preview, humanized errors, cascade audit/event chip |

**Pre-v0.1.25.35 admin servers do not cascade** — operators must manually freeze budgets, revoke keys, and disable webhooks before or after closing the tenant.

## The two rules

### Rule 1 — Close Cascade (server-issued)

On any `* → CLOSED` tenant transition (via `PATCH /v1/admin/tenants/{id}` or `POST /v1/admin/tenants/bulk-action` with `action=CLOSE`), the server drives each owned object into its nearest terminal state:

| Owned object | Terminal state | Notes |
|---|---|---|
| `BudgetLedger` | `CLOSED` | Stamps `closed_at`; drains any outstanding `reserved` back to `remaining`; preserves the final balance snapshot for audit. |
| `ApiKey` | `REVOKED` | Stamps `revoked_at`. |
| Open `Reservation` | `RELEASED` (reason `tenant_closed`) | No overage debt recorded. |
| `WebhookSubscription` | `DISABLED` | Re-enable is blocked by Rule 2 below, making `DISABLED` effectively-terminal for closed owners without adding a new enum value. |

**Ordering.** Ordering is a Mode A concern. Within Mode A's single transaction, the spec says the order SHOULD be:

1. Drain open reservations
2. Close budgets
3. Disable webhooks and revoke API keys (any order)
4. Flip `tenant.status` to `CLOSED` last

Mode B (see below) inverts this by design — the tenant flip commits **first**, and children converge afterward under the Rule 2 guard. Since runcycles' reference server implements Mode B, do not rely on this ordering in practice.

**Audit and event emission.** One record per mutated owned object. The emitted **Event rows** share a server-composed `correlation_id` of the form `tenant_close_cascade:<tenant_id>:<request_id>` — query `GET /v1/admin/events?correlation_id=...` to reconstruct the cascade. **Audit rows** carry `request_id`/`trace_id` (the AuditLogEntry schema has no correlation field); join them via the originating request's `request_id`. The dotted `*_via_tenant_cascade` names are emitted as Event `event_type`s (declared in the governance spec's `EventType` enum since document revision v0.1.25.35, so cascade Events validate against `Event.event_type` and are filterable like any other lifecycle event; Event emission is SHOULD-level, while the audit entries below are a MUST); the matching audit rows are written as `operation="tenant_close_cascade"` with `resource_type`/`resource_id` identifying the mutated object. Reserved dotted names:

- `budget.closed_via_tenant_cascade`
- `webhook.disabled_via_tenant_cascade`
- `api_key.revoked_via_tenant_cascade`
- `reservation.released_via_tenant_cascade`

Servers should additionally emit one Event (dispatchable to webhook subscribers) per mutated owned object using the same dotted kind as its `event_type`. `reservation.released_via_tenant_cascade` is a **ledger-level aggregate**: reservation objects live on the runtime plane, so the admin plane emits one event per closed budget whose `reserved > 0` at close time — identified by `ledger_id`, not an individual reservation — carrying the drained amount as `released_amount`.

See [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol) for how these land on webhook deliveries.

**Idempotency.** Re-issuing close on an already-CLOSED tenant is a no-op *at the tenant level* (returns the current state). Under Mode A no child work remains; under Mode B a re-close completes any outstanding child transitions — without emitting duplicate audit or event rows for already-terminal children. Operator-issued re-close is in fact one of the spec-sanctioned convergence mechanisms for an interrupted Mode B cascade.

### Mode A vs Mode B

The spec (v0.1.25.31) permits two cascade modes:

- **Mode A — Atomic Cascade (preferred).** All owned-object terminal transitions and the tenant flip commit in a single transaction. Rollback on any failure. Strongest guarantee but requires a transactional store.
- **Mode B — Flip-First with Guarded Cascade (conformant alternative).** Tenant flip to `CLOSED` commits first, making Rule 2 active; server then drives children to terminal states inline or via a reconciler. Valid only when: (a) Rule 2 activates at/before flip durability, (b) cascade is idempotent, (c) eventual convergence is guaranteed within a documented bound, (d) observable reads of non-terminal children of a CLOSED tenant remain consistent with stored status until cascade reaches them.

Both modes deliver the same client-observable contract: once the tenant is `CLOSED`, admin-plane mutations against its owned objects return `409 TENANT_CLOSED` regardless of whether the per-object state has flipped yet.

**runcycles' reference server uses Mode B** — backed by Redis, not a transactional database. Operators should not rely on atomic visibility of all child transitions; instead rely on Rule 2.

### Rule 2 — Terminal-Owner Mutation Guard

Every mutating admin-plane operation on an owned object whose parent tenant is `CLOSED` MUST reject with:

```http
HTTP 409 Conflict
Content-Type: application/json

{
  "error": "TENANT_CLOSED",
  "message": "Tenant <tenant_id> is closed; <object_type> is read-only.",
  "request_id": "req-...",
  "trace_id": "..."
}
```

GET endpoints remain available — closed-tenant state is still readable post-mortem for audit and compliance.

### Operations that guard

The spec's Rule 2 scopes the guard to **every mutating admin-plane operation** whose target resource has an owning tenant. Its enumeration (explicitly non-exhaustive) covers:

**Budget plane:**
- `POST /v1/admin/budgets/freeze`
- `POST /v1/admin/budgets/unfreeze`
- `POST /v1/admin/budgets/fund`
- `PATCH /v1/admin/budgets?scope=&unit=` (updateBudget)
- `POST /v1/admin/budgets/bulk-action` (per-row)

**Policy plane (tenant-scoped policies):**
- `POST /v1/admin/policies` (createPolicy)
- `PATCH /v1/admin/policies/{policy_id}` (updatePolicy)

**API key plane:**
- `POST /v1/admin/api-keys` (createApiKey)
- `PATCH /v1/admin/api-keys/{key_id}` (updateApiKey)
- `DELETE /v1/admin/api-keys/{key_id}` (revokeApiKey)

**Webhook plane (admin and tenant self-service paths):**
- `POST /v1/admin/webhooks`, `PATCH`, `DELETE`, `POST .../test`
- `POST /v1/webhooks`, `PATCH`, `DELETE`, `POST .../test`
- `POST /v1/admin/webhooks/{id}/replay`
- `POST /v1/admin/webhooks/bulk-action` (per-row)

**Bulk-action per-row semantics.** On bulk-action endpoints, rows targeting a closed tenant go into the `failed[]` bucket with `error_code=TENANT_CLOSED` — they don't abort the rest of the batch.

### What the runtime plane sees

**Spec (normative):** Rule 2's scope explicitly includes runtime reservation mutations — "any reservation create/commit/release/extend" — so a conformant server MUST reject them with `409 TENANT_CLOSED` once the CLOSED flip is durable. Runtime spec revision v0.1.25.13 binds this directly on the runtime plane: `TENANT_CLOSED` is now part of the runtime `ErrorCode` enum, with a normative closed-tenant binding in the runtime spec's ERROR SEMANTICS. `cycles-server` 0.1.25.47 implements it — the reference-implementation gap this section previously documented is closed. Shipped behavior:

- **Persisting mutations → `409 TENANT_CLOSED`.** Reservation create (`dry_run` absent or `false`), commit, release, and extend against a `CLOSED` owning tenant return `409` with `error=TENANT_CLOSED` once the flip is durable. The check runs inside the same Lua scripts as the budget mutations, so a post-flip request can never partially succeed, and it is not subject to any config-cache TTL.
- **Precedence.** For non-replay mutations, `TENANT_CLOSED` takes precedence over the reservation-state errors (`RESERVATION_FINALIZED`, `RESERVATION_EXPIRED`) — Rule 2 rejects regardless of the child's own current status. Same-key replays of mutations that succeeded before the close are the exception: they retain replay precedence and return the original stored response.
- **Non-persisting evaluations never 409.** A fresh (non-replay) `dry_run=true` create or `POST /v1/decide` on a closed tenant returns `200` with `decision=DENY` and `reason_code=TENANT_CLOSED` — dry-run and decide outcomes are attestations of what live execution would do (and may be captured as signed CyclesEvidence), so they reflect the closed tenant as-if-live instead of erroring. Same-key replays of pre-close evaluations return their original payload.
- **Fail-closed on malformed tenant records.** A tenant record that is present but whose status cannot be determined (undecodable JSON, non-object, missing or non-string `status`, unknown status string) returns `500 INTERNAL_ERROR` before any mutation — on the non-persisting surface too, because the server cannot attest against corrupt governance state. A subject tenant with **no** tenant record at all (runtime-only deployments without a governance plane) is not guarded.
- **Evidence receipts.** A mutation-surface `409 TENANT_CLOSED` on the evidence endpoints — persisting create, commit, release — emits an `error` CyclesEvidence envelope and stamps `cycles_evidence` on the response, like the other live denial codes (extend is not an evidence endpoint).
- **Reads unaffected.** `GET /v1/reservations` and `GET /v1/reservations/{id}` keep working on a closed tenant's reservations for post-close audit, mirroring Rule 2's read-access rule.

**Observability note:** the close cascade revokes the tenant's API keys, and the runtime auth filter rejects CLOSED-tenant keys per request — so tenant-key calls usually still fail with `401 UNAUTHORIZED` before the guard is consulted. In practice `409 TENANT_CLOSED` surfaces in two places: **admin-on-behalf-of release** — of the four guarded mutations, release is the only one the runtime plane exposes to `X-Admin-API-Key` (the admin dual-auth allowlist covers reservation list/get/release only, so create/commit/extend are never admin-key reachable) — and the **post-flip/pre-revocation race window** for tenant-key requests on any of the four.

**Version scope:** `cycles-server` 0.1.25.46 and earlier surface closed tenants on the runtime plane only as `401`s (revoked/rejected keys) or budget-state errors (`BUDGET_CLOSED` on cascaded budgets). Client code on those versions should treat "tenant was closed" as a `401`/`BUDGET_CLOSED` scenario; on 0.1.25.47+ handle `409 TENANT_CLOSED` as well.

## Operator recipe — closing a tenant

```bash
# 1. Preview what will cascade — confirms intent before the irreversible close.
#    The Tenant object itself carries no child counts, so use the list endpoints:
curl -s http://localhost:7979/v1/admin/tenants/acme-corp \
  -H "X-Admin-API-Key: $ADMIN_KEY" | jq '{tenant_id, status}'

# Budgets that will be closed (reserved > 0 will emit released_via_tenant_cascade)
curl -s "http://localhost:7979/v1/admin/budgets?tenant_id=acme-corp" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  | jq '.ledgers[] | {scope, unit, status, reserved}'

# Open reservations that will be released (admin key requires the tenant filter)
curl -s "http://localhost:7979/v1/reservations?tenant=acme-corp&status=ACTIVE" \
  -H "X-Admin-API-Key: $ADMIN_KEY" | jq '.reservations[]'

# 2. Close the tenant — cascade runs automatically
curl -X PATCH http://localhost:7979/v1/admin/tenants/acme-corp \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"status": "CLOSED"}'

# 3. Verify the cascade audit entries
curl -s "http://localhost:7979/v1/admin/audit/logs?tenant_id=acme-corp&operation=tenant_close_cascade" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  | jq '.logs[] | {operation, resource_type, resource_id}'
```

If the `/admin/overview` dashboard still shows frozen budgets on the closed tenant after a few seconds, your admin server is on a pre-v0.1.25.35 version — the cascade hasn't shipped and you need to upgrade. See the [Admin API Guide — Tenant close and cascade semantics](/admin-api/guide).

## Dashboard behavior

The [Cycles Admin Dashboard](/quickstart/deploying-the-cycles-dashboard) (v0.1.25.43+) surfaces cascade behavior:

- **Closed-tenant banner.** Amber read-only banner on `TenantDetailView` when `tenant.status === 'CLOSED'`: "Tenant closed — all owned objects are read-only."
- **CLOSE confirm-dialog preview.** The dialog enumerates what will be terminated: owned budgets, webhook subscriptions, API keys, open reservations, with counts from already-loaded state. "This cannot be undone."
- **`TENANT_CLOSED` humanizer.** Any mutation that races the cascade (stale tab, deep-link, in-flight request) surfaces "Tenant is closed — this object is read-only." instead of the raw 409.
- **Cascade event chip.** Events and audit rows with `_via_tenant_cascade` event-kind suffixes render a small amber "tenant cascade" chip, visually distinguishing cascade-triggered state changes from user-driven ones when operators correlate by `correlation_id`.

See [Using the Cycles Dashboard](/how-to/using-the-cycles-dashboard#closed-tenant-tombstone-and-cascade-preview) for the full UI walkthrough.

## Backward compatibility

- Pre-v0.1.25.35 admin servers do NOT cascade. Operators on older versions must continue manually terminating owned objects before or after the tenant close.
- Pre-v0.1.25.35 servers do NOT return `409 TENANT_CLOSED` — they return the previous per-endpoint error (`409 BUDGET_FROZEN`, `403 FORBIDDEN`, etc.) or may accept mutations against orphaned objects.
- Pre-v0.1.25.36 servers have partial Rule 2 coverage — `.35` guarded budget operations and webhook create/update; `.36` completed policies, api-keys, the remaining webhook mutations, and per-row bulk-action.
- Pre-v0.1.25.43 dashboards render TENANT_CLOSED as a raw 409 error without the humanizer and without the cascade-preview dialog.
- `cycles-server` (runtime) 0.1.25.46 and earlier do NOT return `409 TENANT_CLOSED` on reservation mutations — closed tenants surface there only as `401`s (revoked/rejected keys) or `BUDGET_CLOSED`. The runtime guard ships in 0.1.25.47 (runtime spec v0.1.25.13).

**Re-issuing close on an already-CLOSED tenant** is idempotent at the tenant level across all versions — it returns the current state and emits no duplicate audit entries for already-terminal children. Under Mode B it is not a pure no-op: a re-close completes any outstanding child transitions left by an interrupted cascade.

## Related

- [Error Codes and Error Handling — TENANT_CLOSED](/protocol/error-codes-and-error-handling-in-cycles#tenant-closed-409)
- [Admin API Guide — Tenant close and cascade semantics](/admin-api/guide)
- [Tenant Creation and Management — CLOSED status](/how-to/tenant-creation-and-management-in-cycles#closed)
- [Using the Cycles Dashboard](/how-to/using-the-cycles-dashboard#closed-tenant-tombstone-and-cascade-preview)
- [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol) — cascade event kinds



# Understanding Units in Cycles: USD_MICROCENTS, TOKENS, CREDITS, and RISK_POINTS

Every amount in Cycles — reservations, commits, events, balances — has a unit.

The unit tells the system what is being measured and how to interpret the number. The `unit` field is required on every amount — there is no default unit.

Cycles defines four standard units:

- **USD_MICROCENTS**
- **TOKENS**
- **CREDITS**
- **RISK_POINTS**

Choosing the right unit affects how budgets are expressed, how estimates are calculated, and how the ledger is interpreted.

## USD_MICROCENTS

USD_MICROCENTS is the most precise monetary unit in the protocol.

### Definition

```
1 USD_MICROCENTS = 10⁻⁶ cents = 10⁻⁸ dollars
1 USD = 100 cents = 100,000,000 USD_MICROCENTS
```

### Why microcents?

Model calls are cheap individually. A single GPT-4o-mini call might cost a fraction of a cent.

If the unit were dollars or even cents, many per-call amounts would round to zero. That makes accounting meaningless.

USD_MICROCENTS uses integer arithmetic with enough precision to represent per-call costs without floating point issues.

### Range

The amount field is a 64-bit integer (int64 format) with a minimum of 0. Negative amounts are not valid in standard Amount fields. The SignedAmount variant (used for Balance.remaining) allows negative values to represent overdraft state.

Maximum: `9.22 × 10¹⁸ USD_MICROCENTS ≈ $92.2 billion`

That is more than sufficient for any realistic budget.

### When to use USD_MICROCENTS

- you want to track cost in monetary terms
- you need precision for per-call accounting
- your budgets are expressed in dollars, euros, or other currency (converted to USD_MICROCENTS)
- you want direct correlation between budget state and provider bills

### Example

A model call that costs $0.003:

```
$0.003 = 0.3 cents = 300,000 USD_MICROCENTS
```

Reserve 300,000. If actual usage is 280,000, commit 280,000 and the remaining 20,000 is released automatically.

## TOKENS

TOKENS represents integer token counts, as used by most LLM providers.

### When to use TOKENS

- your budgets are expressed in token counts
- you want direct mapping to provider token metering
- cost varies by model and you want to track consumption in a model-independent unit
- you are budgeting computational capacity rather than monetary cost

### Example

A model call expects to use up to 2,000 input tokens and 500 output tokens.

Reserve 2,500 TOKENS. After the call, actual input was 1,800 and output was 450. Commit 2,250 TOKENS.

### Considerations

Token-based budgeting is simpler but does not account for price differences between models. A budget of 100,000 TOKENS means different monetary costs depending on whether those tokens go to GPT-4o or GPT-4o-mini.

If you need monetary awareness, use USD_MICROCENTS and convert token counts to cost at reservation time.

## CREDITS

CREDITS is a generic integer unit for custom budget systems.

### When to use CREDITS

- your platform defines its own internal currency
- you want to abstract away underlying costs
- different tenants have different pricing and you want to normalize
- you want to decouple budget governance from provider pricing

### Example

A platform defines:

- 1 credit = 1 model call (regardless of model size)
- or 1 credit = some normalized cost unit

Tenants are allocated credits per billing period. Each model call reserves and commits in credits.

### Considerations

Credits require a mapping layer to translate between credits and actual cost. This adds complexity but provides flexibility in pricing and plan design.

CREDITS is a generic integer unit and is optional in v0 implementations — confirm your server supports it before budgeting in credits.

## RISK_POINTS

RISK_POINTS is a generic integer unit for risk-based budgeting.

### When to use RISK_POINTS

- you want to budget side-effect risk rather than cost
- some actions are expensive in risk but cheap in money
- you want to limit how many high-risk actions a tenant or workflow can take
- safety governance is more important than cost governance

### Example

A platform defines:

- read-only model call = 1 risk point
- tool invocation with external API = 5 risk points
- write operation (email, ticket, payment) = 20 risk points

A workflow is allowed 100 risk points per run. This bounds the total side-effect surface regardless of monetary cost.

### Considerations

Risk points are subjective. The team must define what each point represents and calibrate the scale. But for systems where side-effect control matters more than cost control, risk points can be more operationally useful than monetary units.

Like CREDITS, RISK_POINTS is a generic integer unit and is optional in v0 implementations — confirm your server supports it before budgeting in risk points.

## Unit consistency

All amounts within a single reservation lifecycle must use the same unit.

- The reservation estimate, commit actual, and balance amounts must all be in the same unit
- The server returns `400 UNIT_MISMATCH` in three distinct cases:
  - **Reserve and decide** — the `estimate.unit` doesn't match any budget stored for the derived scopes, but at least one of those scopes has a budget in a different unit
  - **Commit** — the `actual.unit` differs from the reservation's `estimate.unit`
  - **Event** — the `actual.unit` doesn't match the budget stored for the target scope
- When the cause is a wrong unit, servers SHOULD populate the error's `details` object with `scope` (where the mismatch was detected), `requested_unit` (what the client sent), and `expected_units` (the units for which a budget does exist), so clients can self-correct without a separate lookup. When no budget exists at any scope in *any* unit, the server returns `404 NOT_FOUND` with the message `"Budget not found for provided scope: ..."` — the runtime plane uses a single `NOT_FOUND` wire code for all resource-not-found conditions, and the message field carries the specific reason. On `/v1/decide` and dry-run reserve, the same "no budget" condition instead surfaces as `200 DENY` with `reason_code=BUDGET_NOT_FOUND`.

This prevents accidental unit confusion (e.g., reserving in tokens and committing in dollars).

Within a balance, all amount fields (remaining, reserved, spent, allocated, debt, overdraft_limit) share the same unit.

## Choosing a unit

A simple decision framework:

### Use USD_MICROCENTS when:
- monetary cost is the primary concern
- you want direct provider bill correlation
- you need per-call precision
- your budgets are expressed in currency

### Use TOKENS when:
- token consumption is the primary metric
- you budget by computational capacity
- you want model-independent counting
- monetary cost varies and you want to decouple

### Use CREDITS when:
- you have a custom platform currency
- you want to abstract provider pricing
- different tenants have different cost structures
- you want plan-based allocation (e.g., 10,000 credits/month)

### Use RISK_POINTS when:
- side-effect control matters more than cost
- you want to bound high-risk actions
- safety governance is a primary goal
- cost and risk are not well correlated

## Multiple units in one system

A single Cycles deployment can use different units for different scopes or action types.

For example:

- tenant budgets in USD_MICROCENTS (monetary ceiling)
- workflow budgets in TOKENS (capacity planning)
- agent budgets in RISK_POINTS (side-effect control)

However, a single reservation lifecycle uses exactly one unit. Multi-unit atomic operations are a v1+ concern.

## Summary

Units define what Cycles is measuring:

- **USD_MICROCENTS** — monetary cost with per-call precision
- **TOKENS** — LLM token consumption
- **CREDITS** — custom platform currency (optional in v0 implementations)
- **RISK_POINTS** — side-effect risk accounting (optional in v0 implementations)

Choosing the right unit depends on whether the primary concern is cost, capacity, pricing abstraction, or safety.

All amounts within a reservation lifecycle and within a balance must use the same unit.

## Next steps

To explore the Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)



# Webhook Event Delivery Protocol

Cycles emits events when budget state changes and delivers them to webhook subscriptions via HTTP POST. This page is the authoritative reference for the delivery protocol.

## Delivery headers

Every webhook delivery includes these HTTP headers:

| Header | Value | Description |
|--------|-------|-------------|
| `Content-Type` | `application/json` | Always JSON |
| `X-Cycles-Signature` | `sha256=<hex>` | HMAC-SHA256 of the raw body using the subscription signing secret. Present whenever the subscription has a signing secret (if no secret was provided when the subscription was created, the server-generated secret is used); omitted when the subscription has no signing secret. |
| `X-Cycles-Event-Id` | `evt_abc123...` | Unique event ID. Use for deduplication. |
| `X-Cycles-Event-Type` | `budget.exhausted` | Dot-notation event type for routing. |
| `X-Cycles-Trace-Id` | `0af7651916cd43dd8448eb211c80319c` | 32-hex W3C Trace Context identifier for the logical operation this event belongs to. Always present on deliveries from v0.1.25.7+ events services. |
| `traceparent` | `00-<trace_id>-<16-hex-span>-<flags>` | W3C Trace Context v00. `trace_id` matches `X-Cycles-Trace-Id`. `span-id` is freshly generated per delivery (not reused from the inbound request). `trace-flags` preserves the inbound sampling decision when the originating request had a valid `traceparent`, otherwise defaults to `01` (sampled). Always present on v0.1.25.7+ events services. |
| `X-Request-Id` | `req-abc-123` | Present when the originating event carries `request_id`. Narrows to side effects of one HTTP request (vs. `X-Cycles-Trace-Id` which may span many). v0.1.25.7+ events services. |
| `User-Agent` | `cycles-server-events/0.1.25.x` | Service identifier and version. Exact patch suffix tracks the shipped events-service build. |
| Custom headers | Per subscription | From the subscription's `headers` map. |

See [Correlation and Tracing](/protocol/correlation-and-tracing-in-cycles) for the full contract on `trace_id` precedence and propagation.

## Payload format

The body is a JSON-serialized Event object:

```json
{
  "event_id": "evt_a1b2c3d4e5f6",
  "event_type": "budget.exhausted",
  "category": "budget",
  "timestamp": "2026-04-01T12:00:00Z",
  "tenant_id": "acme-corp",
  "scope": "tenant:acme-corp/workspace:prod",
  "source": "cycles-admin",
  "actor": {
    "type": "api_key",
    "key_id": "key_abc123",
    "source_ip": "10.0.1.50"
  },
  "data": {
    "ledger_id": "led_xyz",
    "scope": "tenant:acme-corp/workspace:prod",
    "unit": "TOKENS",
    "allocated": 10000,
    "remaining": 0,
    "spent": 10000
  },
  "correlation_id": "3f2a9c14e0b7d5a1",
  "request_id": "req_789",
  "trace_id": "0af7651916cd43dd8448eb211c80319c",
  "metadata": {}
}
```

Fields `scope`, `actor`, `data`, `correlation_id`, `request_id`, `trace_id`, and `metadata` are optional (omitted when null).

**Correlation fields.** `request_id` narrows to one HTTP request; `trace_id` (32-hex W3C) joins related requests when the caller propagates the same context. `correlation_id` is server-managed: the runtime YAML requires deterministic event-cluster hashes, but the current reference runtime leaves the field absent on its implemented emits; selected admin operations populate explicit IDs such as `webhook_create:<id>` and `webhook_bulk_action:<action>:<request_id>`. See [Correlation and Tracing](/protocol/correlation-and-tracing-in-cycles).

## Event types (51)

The current v0.1.25 Admin API `EventType` enum registers 51 event types across seven categories: budget (17), reservation (6), tenant (6), api_key (7), policy (3), webhook (7), and system (5). Implementations may add future event types, and consumers should ignore unrecognized values gracefully. The per-category tables below list the 47 non-cascade types; the four `*_via_tenant_cascade` types (one each in the budget, reservation, api_key, and webhook categories, added to the enum in governance revision v0.1.25.35) are covered in [Tenant-close cascade fan-out](#tenant-close-cascade-fan-out).

Registration does not guarantee emission. The trigger tables describe each registered type's contract; consult the [Event Payloads Reference](/protocol/event-payloads-reference) for the current reference-service emission matrix before subscribing.

::: info Count note
The 51-type / 7-category count tracks the admin OpenAPI enum. The runtime spec's webhook-event guidance section in `cycles-protocol-v0.yaml` lists 35 event types across 6 categories — it predates the `webhook` lifecycle category and some later enum additions.
:::

### Budget events (16)

| Event Type | Trigger |
|------------|---------|
| `budget.created` | Budget ledger created |
| `budget.updated` | Budget ledger configuration changed |
| `budget.funded` | CREDIT funding operation |
| `budget.debited` | Budget debited (funds removed) |
| `budget.reset` | Budget resized (`allocated` changed; `spent`/`reserved`/`debt` preserved) |
| `budget.reset_spent` | New billing period started (`allocated` set; `spent` cleared or explicitly set; `reserved`/`debt` preserved) |
| `budget.debt_repaid` | Outstanding debt repaid |
| `budget.frozen` | Budget set to FROZEN status (no new reservations) |
| `budget.unfrozen` | Budget restored to ACTIVE from FROZEN |
| `budget.closed` | Budget permanently closed (operator action) |
| `budget.threshold_crossed` | Utilization crossed a configured threshold (e.g., 80%, 95%) |
| `budget.exhausted` | Remaining budget reached zero |
| `budget.over_limit_entered` | Debt exceeded overdraft limit |
| `budget.over_limit_exited` | Debt dropped below overdraft limit |
| `budget.debt_incurred` | New debt created by an ALLOW_WITH_OVERDRAFT commit or direct debit |
| `budget.burn_rate_anomaly` | Spend rate exceeds baseline multiplier within the configured window |

### Reservation events (5)

| Event Type | Trigger |
|------------|---------|
| `reservation.denied` | A dry-run reservation or `/v1/decide` evaluation returned `DENY`; current live 4xx reservation errors do not emit this event |
| `reservation.denial_rate_spike` | Denial rate exceeded threshold within window |
| `reservation.expired` | Reservation TTL expired without commit or release |
| `reservation.expiry_rate_spike` | Expiry rate exceeded threshold within window |
| `reservation.commit_overage` | Commit actual exceeded reserved estimate |

### Tenant events (6)

| Event Type | Trigger |
|------------|---------|
| `tenant.created` | New tenant provisioned |
| `tenant.updated` | Tenant configuration changed |
| `tenant.suspended` | Tenant set to SUSPENDED (blocks new reservations) |
| `tenant.reactivated` | Tenant restored to ACTIVE from SUSPENDED |
| `tenant.closed` | Tenant permanently closed |
| `tenant.settings_changed` | Tenant settings (TTL, overage policy, etc.) modified |

### API key events (6)

| Event Type | Trigger |
|------------|---------|
| `api_key.created` | New API key generated |
| `api_key.revoked` | API key permanently revoked (operator action) |
| `api_key.expired` | API key reached its expiration date |
| `api_key.permissions_changed` | API key permissions modified |
| `api_key.auth_failed` | Authentication attempt with invalid key |
| `api_key.auth_failure_rate_spike` | Auth failure rate exceeded threshold within window |

### Policy events (3)

| Event Type | Trigger |
|------------|---------|
| `policy.created` | New policy rule created |
| `policy.updated` | Policy configuration changed |
| `policy.deleted` | Policy removed |

### Webhook events (6)

| Event Type | Trigger |
|------------|---------|
| `webhook.created` | Webhook subscription created |
| `webhook.updated` | Webhook subscription configuration changed |
| `webhook.paused` | Webhook subscription paused by an operator |
| `webhook.resumed` | Webhook subscription resumed by an operator |
| `webhook.disabled` | Webhook subscription auto-disabled after delivery failures |
| `webhook.deleted` | Webhook subscription deleted |

### System events (5)

| Event Type | Trigger |
|------------|---------|
| `system.store_connection_lost` | Redis connection failed |
| `system.store_connection_restored` | Redis connection recovered |
| `system.high_latency` | Operation latency exceeded threshold |
| `system.webhook_delivery_failed` | Webhook delivery permanently failed |
| `system.webhook_test` | Test webhook sent via POST /v1/admin/webhooks/{id}/test |

### Tenant-accessible events

A webhook subscription **owned by a concrete tenant** can only carry — and can only receive — **tenant-accessible** event classes: `budget.*`, `reservation.*`, and `tenant.*` (29 of the 51 registered event types, including the `budget.*` and `reservation.*` cascade fan-out events the admin server emits on tenant close — see the next section). The **admin-only** classes — `api_key.*`, `policy.*`, `webhook.*`, and `system.*` — belong on subscriptions owned by the operator (the `__system__` owner), never on a tenant-owned row.

This is **governance WEBHOOK SUBSCRIPTION INVARIANT 2** (normative, cross-plane, spec revisions v0.1.25.38–.41): a subscription whose owning `tenant_id` is present and `!= "__system__"` MUST NOT carry an admin-only event type or category — *by any provisioning mechanism*. The invariant is a property of the **owning tenant**, not of the caller or the endpoint, so it holds under tenant self-service auth, admin-key, and admin-on-behalf-of alike. The rationale is confidentiality: the owning tenant controls that subscription's delivery URL and signing secret, and `event_categories` is additive with `event_types` in delivery matching, so an admin-only selector on a tenant-owned row would leak admin governance/security telemetry (`api_key` / `policy` / `webhook` / `system` events) to a tenant-controlled endpoint.

::: warning Enforced at three layers (issue #209)
The guarantee is defense-in-depth across the two services — verify your fleet is fully upgraded:
- **Write** — both provisioning planes reject an admin-only type or category on a concrete-tenant subscription with `400 INVALID_REQUEST`. The tenant self-service plane (`POST /v1/webhooks`, `PATCH /v1/webhooks/{subscription_id}`) since **cycles-server-admin 0.1.25.50** (governance v0.1.25.38); the admin plane (`POST /v1/admin/webhooks?tenant_id=X`, `PATCH /v1/admin/webhooks/{id}`) since **0.1.25.51** (governance v0.1.25.40) — update validates the *effective resulting* selectors (each array as it stands after the update — the request's value where provided, the stored value where omitted; `PATCH` replaces a supplied array, it does not merge), so a status-only reactivation validates the still-stored selectors and can't re-enable a disabled offender that holds admin-only ones. `__system__`-owned subscriptions are exempt (system-wide monitoring is legitimate).
- **Dispatch** — since **0.1.25.51**, live dispatch and replay skip any admin-only event per-event for a concrete-tenant subscription, fail-closed and independent of stored-selector correctness (a default-on startup reconciler additionally strips legacy admin-only selectors from stored non-`DISABLED` rows — best-effort hygiene, see below).
- **Last-mile delivery** — since **cycles-server-events 0.1.25.23**, the delivery worker re-checks the boundary immediately before every outbound POST (initial, retry, and recovered redeliveries), catching deliveries queued before the upgrade and every retry. **Rolling-deploy caveat:** this is a per-worker guarantee — airtight only once *all* delivery workers are on 0.1.25.23.

**Operators upgrading past 0.1.25.49 (admin) / 0.1.25.22 (events)** should audit existing tenant subscriptions for admin-only selectors. Confidentiality does not depend on the audit — the fail-closed dispatch and last-mile boundaries already withhold the events. As storage hygiene, 0.1.25.51 also runs a **default-on, best-effort startup reconciler** (`webhook.category-boundary.reconcile-on-startup`, default `true`) that strips admin-only selectors from stored non-`DISABLED` concrete-tenant rows and disables empty-both rows; it is not the security mechanism (dispatch already withholds the events, and a `DISABLED` row delivers nothing). The [0.1.25.50 release notes](https://github.com/runcycles/cycles-server-admin/releases/tag/v0.1.25.50) carry a `redis-cli`/`jq` recipe to find offenders manually.
:::

#### Monitoring a specific tenant's admin-only events

Because a tenant-owned subscription can no longer carry admin-only classes, per-tenant admin monitoring (e.g. one tenant's `api_key.*` or `policy.*` events pointed at an operator endpoint) moves to a **`__system__`-owned** subscription — create it via `POST /v1/admin/webhooks` with **no `tenant_id`** parameter. A `__system__`-owned subscription may carry admin-only selectors, so name the admin event types you want in `event_types` (create requires a non-empty `event_types` per INVARIANT 1); to cover whole admin categories, pair a representative admin type with `event_categories` (e.g. `event_types: ["api_key.created"]`, `event_categories: ["api_key", "policy"]`). Because `__system__` is in the dispatch union for every tenant, that subscription receives those admin events for **all** tenants; select the tenant you care about **client-side** on the delivered envelope's `tenant_id`.

`scope_filter` generally can't do the per-tenant narrowing here: `api_key.*`, `webhook.*`, and `system.*` events are **null-scoped**, and a `scope_filter` excludes null-scoped events, so it would deliver none of them. The one exception is `policy.*`, which carries a real tenant-bounded `scope` and so *can* be `scope_filter`ed if you only need policy events. For everything else, client-side `tenant_id` filtering is the general solution. The `__system__` row is operator-owned, so its URL and secret stay operator-controlled.

::: tip `/test` probe exception
The owner-triggered webhook test (`POST /v1/webhooks/{id}/test` and its admin twin) POSTs a single synthetic `system.webhook_test` connectivity event **directly** to the subscription's own endpoint, bypassing the dispatch queue. A tenant-owned subscription MAY receive its own test probe even though `system.webhook_test` is a `system.*` (admin-only) type — the payload is an owner-requested `{subscription_id, test:true}` ping carrying no governance telemetry, and the subscription's stored selectors are unchanged and still must satisfy INVARIANT 2. This exception is limited to the synthetic test event on the `/test` operations; no real `system.*`/`api_key.*`/`policy.*`/`webhook.*` event from the event stream reaches a tenant-owned subscription.
:::

### Tenant-close cascade fan-out

The admin server emits cascade fan-out events with the `_via_tenant_cascade` suffix as side effects of a `* → CLOSED` tenant transition (Rule 1 — Close Cascade). All four names are declared in the governance spec's `EventType` enum since document revision v0.1.25.35, so they count toward the 51 registered types and are filterable like any other lifecycle event. Emission is SHOULD-level in the spec (the matching per-object audit entries are a MUST), so non-reference servers may not emit them — keep ignoring unrecognized event types gracefully:

- `budget.closed_via_tenant_cascade` — one per owned `BudgetLedger`.
- `reservation.released_via_tenant_cascade` — a **ledger-level aggregate**: one per closed budget with `reserved > 0`, carrying `released_amount`. Reason `tenant_closed`; no overage debt.
- `api_key.revoked_via_tenant_cascade` — one per owned API key.
- `webhook.disabled_via_tenant_cascade` — one per owned webhook subscription.

All four carry a server-composed `correlation_id` of the form `tenant_close_cascade:<tenant_id>:<request_id>`, letting subscribers correlate cascade side effects to the operator action that triggered them (audit rows for the operation join via `request_id`/`trace_id`). The dashboard (v0.1.25.43+) renders a "tenant cascade" chip on audit and event-timeline rows with these suffixes.

See [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics) for the full Rule 1 / Rule 2 contract and Mode A / Mode B semantics.

## Delivery status lifecycle

<DeliveryStateMachine />

| Status | Meaning |
|--------|---------|
| `PENDING` | Queued for delivery, not yet attempted |
| `SUCCESS` | Delivered and received HTTP 2xx response |
| `RETRYING` | Failed but retries remain, scheduled for retry |
| `FAILED` | All retries exhausted or delivery expired |

## Retry policy

Failed deliveries are retried with exponential backoff:

```
delay = min(initial_delay_ms * backoff_multiplier ^ (attempt - 1), max_delay_ms)
```

| Setting | Default | Description |
|---------|---------|-------------|
| `max_retries` | 5 | Maximum retry attempts (6 total including first attempt) |
| `initial_delay_ms` | 1000 | Delay before first retry |
| `backoff_multiplier` | 2.0 | Multiplier applied per retry |
| `max_delay_ms` | 60000 | Maximum delay cap |

**Default retry schedule:** 1s, 2s, 4s, 8s, 16s (capped at 60s).

**Success criteria:** HTTP response status 200–299.

### Auto-disable

After `disable_after_failures` (default 10) consecutive delivery failures, the subscription status is set to `DISABLED`. The counter resets to 0 on any successful delivery. Disabled subscriptions must be manually re-enabled via `PATCH /v1/admin/webhooks/{id}`.

### Stale delivery handling

Deliveries older than `MAX_DELIVERY_AGE_MS` (default 24 hours) are automatically marked FAILED without attempting HTTP delivery. This prevents delivering stale events after a prolonged events service outage.

## Transport

Outbound webhook deliveries negotiate **HTTP/1.1 only** (no HTTP/2 / h2c). This was pinned in `cycles-server-events` v0.1.25.5 to close a silent body-drop bug against HTTP/2 reverse proxies that upgrade `http://` to h2c (closes `cycles-server-events#16`). Receivers behind HTTP/1.1-only proxies were unaffected; receivers behind HTTP/2-capable proxies gain consistent body delivery.

Response bodies are discarded (`HttpResponse.BodyHandlers.discarding()`) so large responses from misbehaving receivers don't pin memory.

## Signature verification

The `X-Cycles-Signature` header contains `sha256=<hex>` where `<hex>` is the HMAC-SHA256 of the raw JSON request body using the subscription's signing secret as the key.

**Verification steps:**

1. Read the raw request body as bytes (do not parse JSON first)
2. Compute HMAC-SHA256 using your copy of the signing secret
3. Compare `sha256=<computed_hex>` with the `X-Cycles-Signature` header using a constant-time comparison
4. Reject the request if they do not match

See [Webhook Integrations](/how-to/webhook-integrations#signature-verification) for implementation in Python, Node.js, Go, and Java.

## At-least-once delivery

Webhooks are delivered at least once. Duplicates can occur due to:

- Network retries (timeout before response received, but server processed it)
- Events service restart during delivery
- Event replay operations

**Deduplication:** Use the `X-Cycles-Event-Id` header as a deduplication key. Store processed event IDs with a short TTL (24h recommended) and skip events you have already seen.

## Redis keys

The events service uses these Redis data structures (shared with the admin server):

| Key | Type | Written By | Read By | Description |
|-----|------|-----------|---------|-------------|
| `dispatch:pending` | LIST | Admin (LPUSH) | Events (BLMOVE) | Delivery IDs awaiting processing |
| `dispatch:processing` | LIST | Events (BLMOVE) | Events (LREM / recovery) | Claimed delivery IDs retained until acknowledged |
| `dispatch:processing:claimed_at` | ZSET | Events | Events | Claim timestamps used for idle-gated crash recovery |
| `dispatch:processing:claim_owner` | HASH | Events | Events | Per-delivery claim-generation token that prevents a stale worker from acknowledging a successor's claim |
| `dispatch:ordering:lock` | STRING | Events | Events | Renewable owner token for the cross-replica claim/send critical section |
| `dispatch:retry` | ZSET | Events (ZADD) | Events (ZRANGEBYSCORE) | Retry queue (score = timestamp) |
| `dispatch:failed` | LIST | Events (LPUSH/LTRIM) | Operators | Bounded quarantine for corrupt delivery records |
| `delivery:{id}` | STRING | Admin (SET) | Events (GET/SET) | Delivery record JSON (14-day TTL) |
| `event:{id}` | STRING | Admin (SET) | Events (GET) | Event record JSON (90-day TTL) |
| `webhook:{id}` | STRING | Admin (SET) | Events (GET/SET) | Subscription JSON |
| `webhook:secret:{id}` | STRING | Admin (SET, encrypted) | Events (GET, decrypts) | AES-256-GCM encrypted signing secret |

## Next steps

- [Webhook Integrations](/how-to/webhook-integrations) — PagerDuty, Slack, ServiceNow examples with signature verification code
- [Managing Webhooks](/how-to/managing-webhooks) — create, update, test, and replay webhooks
- [Deploying the Events Service](/quickstart/deploying-the-events-service) — setup and configuration
- [Cycles Security](/security#webhook-security) — SSRF protection, encryption, and at-least-once delivery



# Webhook Scope Filter Syntax

Webhook subscriptions can filter events by scope path using the `scope_filter` field. When set, only events whose `scope` matches the filter are delivered to your endpoint.

::: warning Historical implementation divergence — cycles-server-admin 0.1.25.48 and earlier
**Resolved in current releases:** cycles-server-admin **0.1.25.49** (2026-07-10) ships the spec-conformant admin matcher (and applies the same filter on the replay path), and cycles-server **0.1.25.47** refined the runtime matcher's two edge cases to the same semantics (blank/whitespace-only event scopes are treated as unscoped and excluded from filtered subscriptions; trailing-`/*` filters require a non-empty child segment). The two matchers are pinned to the same table of (filter, scope, expected) test cases, so on admin 0.1.25.49+ / runtime 0.1.25.47+ both planes match identically per the spec and cannot drift. The rest of this callout is history for deployments on older versions.

**Scope of the divergence (admin 0.1.25.48 and earlier):** it applies only to **admin-plane-emitted events** (tenant, api_key, policy, webhook lifecycle, and admin-initiated budget events) plus **replay**. Runtime-emitted events (reservations, runtime budget events — the bulk of webhook volume) have always been matched by the runtime server's own matcher, which already implemented the spec semantics below (exact match, trailing-`*` prefix, null scope excluded; the two edge cases refined in 0.1.25.47 — see [Edge cases](#edge-cases)). In other words: on admin 0.1.25.48 and earlier, the *same filter* matched differently depending on which plane emitted the event.

The admin OpenAPI spec (normative, and described first below) defines exact-match semantics with an optional trailing `*` wildcard. The **admin server's** matcher (`WebhookRepository.matchesScope`, 0.1.25.48 and earlier) instead did **literal prefix matching**: a blank filter matches everything, a null event scope always matches, and otherwise the event scope must `startsWith(scope_filter)` — with a bare `"*"` filter special-cased to match everything. Three practical consequences for admin-plane events on those versions:

1. **Trailing-`/*` filters match no admin-plane events.** The admin matcher compares the `*` literally, and real scopes never contain a `*` — so a spec-form filter delivers runtime events but silently misses admin-plane events until 0.1.25.49.
2. **A filter without `*` is a prefix, not an exact match.** `tenant:acme-corp/workspace:prod` also matches `tenant:acme-corp/workspace:prod/workflow:support` (and even `tenant:acme-corp/workspace:prod-eu`, since matching is character-wise). End the filter with `/` to bound it to child scopes.
3. **Events with a null scope ARE delivered** to scope-filtered subscriptions (a null scope matches every filter), rather than being excluded.

**Recommendation:** write filters in the spec's `/*` form. It matches runtime-emitted events on every version and both planes on admin 0.1.25.49+ / runtime 0.1.25.47+. There is no single filter form that matches child scopes on both planes on admin 0.1.25.48 and earlier: `/*` misses admin events, bare-prefix misses runtime events. If you must catch both on those older versions, subscribe without a `scope_filter` and filter client-side on the envelope `scope`. Upgrading to 0.1.25.49 is a **behavior change** for existing prefix-style filters — see the [release's migration notes](https://github.com/runcycles/cycles-server-admin/releases/tag/v0.1.25.49) (bare-prefix filters must be rewritten as `…/*`; "base + descendants" coverage now needs two subscriptions).
:::

## Matching rules (spec semantics — normative)

Per the admin OpenAPI spec, the scope filter supports two modes:

### Exact match (no wildcard)

The event scope must exactly equal the filter string.

```json
{
  "scope_filter": "tenant:acme-corp/workspace:prod"
}
```

Under spec semantics this delivers events **only** when the event scope is exactly `tenant:acme-corp/workspace:prod`; events scoped to `tenant:acme-corp/workspace:prod/workflow:support` would **not** match. **Admin plane, 0.1.25.48 and earlier:** this filter is treated as a prefix, so child-scope admin events *do* match (consequence 2 above). The runtime matcher applies the exact-match spec semantics.

### Prefix match (trailing wildcard)

A filter ending with `*` matches any event scope that starts with the prefix before the `*`.

```json
{
  "scope_filter": "tenant:acme-corp/*"
}
```

Under spec semantics this delivers events for any scope under `tenant:acme-corp/`, including:
- `tenant:acme-corp/workspace:prod`
- `tenant:acme-corp/workspace:prod/workflow:support`
- `tenant:acme-corp/workspace:staging/agent:bot-1`

**Per plane (0.1.25.48 and earlier):** the runtime matcher handles this correctly; the admin matcher compares the `*` literally and delivers nothing (consequence 1 above).

### No filter (default)

If `scope_filter` is null, empty, or not provided, the subscription matches **all events** regardless of scope. Both semantics agree on this.

```json
{
  "scope_filter": null
}
```

## Syntax summary

| Filter | Spec semantics (normative; both planes on admin 0.1.25.49+ / runtime 0.1.25.47+) | Admin plane, 0.1.25.48 and earlier (prefix match) |
|---|---|---|
| `null` / empty / blank | All events | All events |
| `tenant:acme-corp` | Only scope exactly `tenant:acme-corp` | Any scope starting with `tenant:acme-corp` (including `tenant:acme-corpX`) |
| `tenant:acme-corp/` | Only scope exactly `tenant:acme-corp/` (unlikely to exist) | Any scope starting with `tenant:acme-corp/` |
| `tenant:acme-corp/*` | Scopes **under** `tenant:acme-corp/` — a non-empty child segment is required from cycles-server 0.1.25.47 onward (the degenerate empty-child scope `tenant:acme-corp/` matched on 0.1.25.46 and earlier) | Nothing (literal `*` never appears in real scopes) |
| `tenant:acme-corp/workspace:prod` | Only that exact scope | That scope and anything starting with it |
| `*` | Undefined by spec; the runtime matcher treats it as an empty-prefix trailing wildcard — any non-blank scope matches from cycles-server 0.1.25.47 onward (on 0.1.25.46 and earlier a blank `""` scope also matched, via the empty-prefix comparison) | All events (including null-scope) |
| *(any filter)* vs. null-scope event | Not delivered | Delivered |

## What's NOT supported

- **Mid-string wildcards** — `tenant:*/workspace:prod` does not work. In the spec, `*` is only meaningful at the end of the filter string; a mid-string `*` is treated as a literal character on both planes (and the pre-fix admin matcher treats even a trailing `*` as literal).
- **Multiple wildcards** — `tenant:acme-corp/*/workflow:*` is not valid.
- **Regex** — no regular expression matching is supported.
- **Glob patterns** — `?`, `[a-z]`, and other glob characters are treated as literal characters.
- **Multiple scope filters per subscription** — each subscription has a single `scope_filter` string. Create multiple subscriptions if you need to watch multiple unrelated scopes.

Under spec semantics — and the runtime matcher — a `*` anywhere other than the end of the filter string is treated as a **literal character** in an exact-match comparison (which almost certainly won't match any real scope). The pre-fix admin matcher (0.1.25.48 and earlier) treats every `*` as literal, including a trailing one.

## Examples

The examples below use the spec `/*` form — correct for runtime-emitted events on every version, and for both planes on cycles-server-admin 0.1.25.49+ / cycles-server 0.1.25.47+. (On admin 0.1.25.48 and earlier, admin-plane events will not match these filters; see the callout above.)

### Subscribe to all events for one tenant

A subscription must match on at least one selector, so at least one of `event_types` / `event_categories` must be non-empty — the server rejects the empty-both state with `400 INVALID_REQUEST` (governance revision v0.1.25.39; enforced since cycles-server-admin 0.1.25.50). The two arrays are additive (union) in delivery matching. Note the create/update asymmetry: `POST /v1/admin/webhooks` (and `/v1/webhooks`) requires a non-empty `event_types` specifically, while `PATCH` may clear `event_types` to empty as long as `event_categories` is non-empty — a **category-only** subscription is valid on update. To cover whole categories on create, pair a representative type with the category list. See [Category-based subscriptions](/how-to/managing-webhooks#category-based-subscriptions).

```bash
curl -X POST http://localhost:7979/v1/admin/webhooks \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://ops.example.com/cycles-events",
    "event_types": ["budget.exhausted"],
    "event_categories": ["budget", "reservation", "tenant"],
    "scope_filter": "tenant:acme-corp/*"
  }'
```

This covers the **tenant-accessible** classes (`budget` / `reservation` / `tenant`) for one tenant. It omits the admin-only categories: `api_key.*`, `webhook.*`, and `system.*` events are **null-scoped**, so a `scope_filter` excludes them — you can't narrow those to one tenant with `scope_filter` (filter client-side on the envelope `tenant_id` instead; see [Tenant-accessible events](/protocol/webhook-event-delivery-protocol#tenant-accessible-events)). The one admin category that *is* scope-filterable is `policy.*` (it carries a real tenant-bounded scope) — add `policy` to `event_categories` if you also want this tenant's policy events. Note this is a `__system__`-owned subscription (admin key, no `tenant_id`); a **tenant-owned** subscription can't carry admin-only categories at all.

### Subscribe to one specific workspace

```bash
curl -X POST http://localhost:7979/v1/admin/webhooks \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://ops.example.com/prod-alerts",
    "event_types": ["budget.exhausted", "reservation.denied"],
    "scope_filter": "tenant:acme-corp/workspace:prod/*"
  }'
```

This delivers only `budget.exhausted` and `reservation.denied` events (runtime-emitted) where the scope starts with `tenant:acme-corp/workspace:prod/`.

### No scope filter — receive everything

```bash
curl -X POST http://localhost:7979/v1/admin/webhooks \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://ops.example.com/all-events",
    "event_types": ["budget.exhausted"],
    "event_categories": ["budget", "reservation", "tenant", "api_key", "policy", "webhook", "system"]
  }'
```

`scope_filter` omitted — this subscription receives matching events from all scopes (including unscoped events). All-categories `event_categories` plus a representative `event_types` entry is the "everything" form on **create**, where `event_types` must be non-empty; on a later `PATCH` the type could be cleared, leaving the all-categories subscription category-only. The two arrays are a union in delivery matching. This is a `__system__`-owned subscription (admin key, no `tenant_id`), so the admin-only categories are permitted; a **tenant-owned** subscription (`/v1/webhooks`, or `/v1/admin/webhooks?tenant_id=X`) may carry only `budget` / `reservation` / `tenant` (governance INVARIANT 2 — see [Tenant-accessible events](/protocol/webhook-event-delivery-protocol#tenant-accessible-events)).

### Combining event type filter with scope filter

Both filters apply with AND logic. An event must match **both** the event type list and the scope filter to be delivered.

```bash
curl -X POST http://localhost:7979/v1/admin/webhooks \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://ops.example.com/cost-alerts",
    "event_types": ["budget.exhausted", "budget.over_limit_entered"],
    "scope_filter": "tenant:acme-corp/workspace:prod/*"
  }'
```

This delivers only `budget.exhausted` **or** `budget.over_limit_entered` events (runtime-emitted) **and** only when the scope starts with `tenant:acme-corp/workspace:prod/`.

## Events without scope

Some events may not have a `scope` field (null). The two semantics differ:

- **Spec semantics (normative):** when `scope_filter` is set and an event has a null scope, the event is **not delivered** to that subscription. Use a separate subscription without a scope filter to capture unscoped events.
- **Per plane:** the runtime matcher follows the spec (null scope excluded from filtered subscriptions — `EventEmitterRepository.matchesScope`); the admin plane on 0.1.25.48 and earlier delivers null-scope events to every filter (fixed in cycles-server-admin 0.1.25.49).

::: tip Note on `reservation.commit_overage`
As of cycles-server v0.1.25.46, `reservation.commit_overage` is emitted **with** the reservation's scope path on the envelope, so it participates in scope filtering like any other scoped event. (Earlier releases emitted it with a null envelope scope, in which case the null-scope rules above applied.)
:::

## Edge cases

- **Whitespace-only filter** (e.g., `"   "`): Treated the same as null — matches all events (both semantics).
- **Filter `"*"` alone**: Under spec semantics this is undefined (arguably an exact match against a scope literally equal to `*`). The runtime matcher treats it as "match any scoped event": from cycles-server 0.1.25.47 onward, blank/whitespace-only scopes are treated as unscoped and excluded — the same semantics as the admin matcher since cycles-server-admin 0.1.25.49; on runtime 0.1.25.46 and earlier, a blank `""` scope still matched via the empty-prefix comparison. Only the admin plane on 0.1.25.48 and earlier matches **all** events including null-scope ones. Prefer omitting `scope_filter` entirely to mean "everything".
- **Blank event scopes** (`""` or whitespace-only): From cycles-server 0.1.25.47 (runtime) and cycles-server-admin 0.1.25.49 onward, blank and null scopes are both treated as unscoped — excluded from every scope-filtered subscription. On runtime 0.1.25.46 and earlier, only `null` was checked (a blank scope could match the bare `*` filter); on admin 0.1.25.48 and earlier, null-scope events match every filter.
- **Empty-child scopes against trailing `/*`** (e.g., event scope `tenant:acme-corp/` against filter `tenant:acme-corp/*`): From cycles-server 0.1.25.47 onward, the runtime matcher requires a non-empty remainder after the prefix — `tenant:acme-corp/` no longer matches (the spec says "all scopes **under** acme-corp"). On 0.1.25.46 and earlier it matched via plain `startsWith`. Unchanged on both: the bare base scope `tenant:acme-corp` (no trailing slash) never matches a `…/*` filter on the runtime plane.

## Related

- [Managing Webhooks](/how-to/managing-webhooks) — creating, updating, and testing subscriptions
- [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol) — delivery mechanics, retry schedule, signatures
- [Event Payloads Reference](/protocol/event-payloads-reference) — payload schemas for all event types

---

# How-To Guides



# Add Cycles with Claude, Codex, Cursor, or Windsurf

This page is for engineers who want their AI coding assistant to integrate Cycles into an existing codebase. Open Claude Code (or Codex, Cursor, Windsurf) in your repo, paste the prompt below, and let it wire one budget-enforced boundary, with a test that proves enforcement, in a single session.

**The invariant: Cycles must run before the costly action on the same execution path.** Every rule, example, and test on this page exists to prove or enforce that one statement.

::: tip If you only need MCP host setup
For Claude Desktop / Claude Code / Cursor / Windsurf MCP server config, see the per-host quickstarts: [Claude Desktop](/quickstart/mcp-claude-desktop) · [Claude Code](/quickstart/mcp-claude-code) · [Cursor](/quickstart/mcp-cursor) · [Windsurf](/quickstart/mcp-windsurf). MCP gives the assistant access to Cycles tools — it does not by itself enforce budgets in your application's execution path. See [MCP vs enforcement](#mcp-availability-is-not-enforcement) below.
:::

## The integration contract

Hand this contract to the assistant. It is the do/don't list that turns "integrate Cycles" from an open-ended task into a bounded one.

```md
You are integrating Cycles into an existing agent codebase.

Goal:
Add pre-execution budget/action enforcement around one LLM call or tool call.
Prove enforcement with a test, then expand.

Do:
1. Identify model calls and external side-effect tool calls in this repo.
2. Pick ONE boundary to wrap first. Highest cost, highest frequency, or
   highest blast radius wins.
3. Use the language-specific client:
   - Python:        `runcycles` package, `@cycles` decorator
   - TypeScript:    `runcycles` package, `withCycles` HOF
   - Java/Spring:   `cycles-client-java-spring`, `@Cycles` annotation
   - Rust:          `runcycles` crate, `with_cycles()` (auto) or
                    `ReservationGuard` (manual / streaming)
4. Read configuration from environment:
   - CYCLES_BASE_URL   (e.g. http://localhost:7878)
   - CYCLES_API_KEY    (cyc_live_... — issued by the Cycles Admin Server)
   - CYCLES_TENANT     (e.g. acme-corp)
5. Add a graceful denial fallback: when budget is denied, do NOT execute the
   downstream call. Return a fallback, downgrade to a cheaper model, or queue.
6. Add ONE test that proves the downstream call is not invoked when Cycles
   denies. Mock the model client; assert it received zero calls on DENY.

Do not:
- Rely on prompts as enforcement.
- Add only logging without a deny path.
- Place the Cycles check after the downstream call.
- Treat MCP tool availability as hard enforcement. The MCP server exposes
  Cycles tools to the assistant; it does not gate the application's own
  execution path. Production enforcement belongs in the SDK wrapper or
  gateway, not in the host's tool list.
- Invent new patterns. Use the decorator / HOF / annotation as documented.
- Wrap more than one boundary in the first pass. Ship one, test it, then
  expand.

Success test:
- One LLM (or external tool) call wrapped with the language-appropriate
  Cycles primitive.
- Env vars read via `CyclesConfig.from_env()` / `CyclesConfig.fromEnv()` /
  Spring properties.
- One test asserting the downstream client is not called on budget denial.
- The change is a small diff — ideally under ~50 lines of production code
  plus the test. Spring and Rust may run slightly longer.
```

## Definition of done

The integration is complete when **all** of these are true. This is the checklist Claude/Codex should optimize toward — and that you should grade the diff against before merging.

- [ ] At least one LLM or external tool call is wrapped with the Cycles primitive.
- [ ] The Cycles `reserve` (or decorator/HOF/annotation entry) runs **before** the downstream call.
- [ ] On `DENY`, the downstream call is **skipped** entirely. No model API request fires.
- [ ] On success, **actual usage is committed** (not just the original estimate).
- [ ] On thrown exception, the reservation is **released** so budget returns to the pool.
- [ ] One test mocks the downstream client and asserts it received zero calls on DENY.
- [ ] `CYCLES_BASE_URL`, `CYCLES_API_KEY`, `CYCLES_TENANT` are read from env, not hardcoded.

If any item is unchecked, the integration is not done — even if the happy path runs.

## Copy this prompt into your assistant

Paste this verbatim into Claude Code, Codex, Cursor, or Windsurf at the root of your repo:

```md
Integrate Cycles into this repo.

First inspect the codebase and identify:
- all LLM calls (OpenAI, Anthropic, Bedrock, Gemini, Groq, Ollama, etc.)
- all tool calls with external cost or real-world side effects
- tenant / user / run identifiers already available in request context

Then implement the smallest safe integration:
- wrap ONE LLM call with Cycles
- reserve budget before execution
- commit actual usage after execution
- release on failure
- deny BEFORE the downstream call when the budget is exhausted
- read CYCLES_BASE_URL, CYCLES_API_KEY, CYCLES_TENANT from environment
- add ONE test proving the downstream call is not made on DENY

Follow the integration contract and constraints from:
https://runcycles.io/how-to/add-cycles-with-claude-or-codex

Use the official docs:
- Quickstart:                https://runcycles.io/quickstart/end-to-end-tutorial
- Existing-app integration:  https://runcycles.io/how-to/adding-cycles-to-an-existing-application
- Python client:             https://runcycles.io/quickstart/getting-started-with-the-python-client
- TypeScript client:         https://runcycles.io/quickstart/getting-started-with-the-typescript-client
- Spring Boot starter:       https://runcycles.io/quickstart/getting-started-with-the-cycles-spring-boot-starter
- Rust client:               https://runcycles.io/quickstart/getting-started-with-the-rust-client
- Error handling (Python):   https://runcycles.io/how-to/error-handling-patterns-in-python
- Error handling (TS):       https://runcycles.io/how-to/error-handling-patterns-in-typescript
- Error handling (Rust):     https://runcycles.io/how-to/error-handling-patterns-in-rust

Pick the language guide that matches this repo. Do NOT wrap more than one
boundary in this pass. Stop and report when the test passes.
```

The full machine-readable index is at <a href="/llms.txt" target="_blank" rel="noopener"><code>/llms.txt</code></a> — most assistants can fetch it to discover the rest of the documentation.

## Drop-in AGENTS.md / CLAUDE.md / .cursorrules snippet

To make the integration recipe survive across sessions, drop this into your repo's `AGENTS.md`, `CLAUDE.md`, or `.cursorrules` file. Future AI sessions inherit the rules without you re-pasting the prompt.

The snippet is published as a static file you can curl:

```bash
curl -O https://runcycles.io/agents/cycles-integration.md
# then append or include in your repo's AGENTS.md / CLAUDE.md / .cursorrules
```

The same content is mirrored at <a href="/agents/cycles-integration.md" target="_blank" rel="noopener"><code>/agents/cycles-integration.md</code></a> (opens in a new tab — the file is served as raw markdown, not a VitePress page).

## Minimum viable integration by language

The assistant should produce a diff that looks like one of the four blocks below. These are lifted directly from the language quickstarts — there is no new pattern here.

::: code-group

```python [Python]
# pip install runcycles
# env: CYCLES_BASE_URL, CYCLES_API_KEY, CYCLES_TENANT
from runcycles import (
    CyclesClient, CyclesConfig, BudgetExceededError, cycles, set_default_client,
)

set_default_client(CyclesClient(CyclesConfig.from_env()))

def estimate_actual(summary: str) -> int:
    # Prefer provider usage/cost metadata (e.g. response.usage.total_tokens)
    # when available. This length-based placeholder keeps the example short.
    return max(1, len(summary) * 5)              # USD_MICROCENTS

@cycles(
    estimate=2_000_000,                          # USD_MICROCENTS — tune from logs
    actual=estimate_actual,
    action_kind="llm.completion",
    action_name="openai:gpt-4o",
)
def generate_summary(document: str) -> str:
    return openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": f"Summarize: {document}"}],
        max_tokens=2000,
    ).choices[0].message.content

def summarize_or_fallback(document: str) -> str:
    try:
        return generate_summary(document)
    except BudgetExceededError:
        return "Summary unavailable — budget limit reached."
```

```typescript [TypeScript]
// npm install runcycles
// env: CYCLES_BASE_URL, CYCLES_API_KEY, CYCLES_TENANT
import {
  CyclesClient, CyclesConfig, BudgetExceededError,
  withCycles, setDefaultClient,
} from "runcycles";

setDefaultClient(new CyclesClient(CyclesConfig.fromEnv()));

const generateSummary = withCycles(
  {
    estimate: 2_000_000,                         // USD_MICROCENTS — tune from logs
    actual: (summary: string) => Math.max(1, summary.length * 5),
    actionKind: "llm.completion",
    actionName: "openai:gpt-4o",
  },
  async (document: string) => {
    const r = await openai.chat.completions.create({
      model: "gpt-4o",
      messages: [{ role: "user", content: `Summarize: ${document}` }],
      max_tokens: 2000,
    });
    return r.choices[0].message.content!;
  },
);

export async function summarizeOrFallback(document: string) {
  try {
    return await generateSummary(document);
  } catch (err) {
    if (err instanceof BudgetExceededError) {
      return "Summary unavailable — budget limit reached.";
    }
    throw err;
  }
}
```

```java [Java / Spring]
// pom.xml: io.runcycles:cycles-client-java-spring
// application.yml: cycles.base-url, cycles.api-key, cycles.tenant
import io.runcycles.client.java.spring.annotation.Cycles;
import io.runcycles.client.java.spring.model.CyclesProtocolException;

@Service
public class SummaryService {

    @Cycles(value = "2000000",
            // Adapt to your client's response shape — the SpEL expression
            // must resolve against whatever generateSummary() returns.
            actual = "#result.usage.totalTokens * 8",
            actionKind = "llm.completion",
            actionName = "openai:gpt-4o")
    public ChatResponse generateSummary(String document) {
        return openAiClient.chat(document);
    }

    public String summarizeOrFallback(String document) {
        try {
            return generateSummary(document).text();
        } catch (CyclesProtocolException e) {
            if (e.isBudgetExceeded()) {
                return "Summary unavailable — budget limit reached.";
            }
            throw e;
        }
    }
}
```

```rust [Rust]
// Cargo.toml: runcycles = "0.3"
// env: CYCLES_BASE_URL, CYCLES_API_KEY, CYCLES_TENANT
use runcycles::{
    with_cycles, CyclesClient, CyclesConfig, Error, WithCyclesConfig,
    models::{Amount, Subject},
};

pub async fn summarize_or_fallback(
    client: &CyclesClient,
    document: &str,
) -> String {
    let result = with_cycles(
        client,
        WithCyclesConfig::new(Amount::usd_microcents(2_000_000))
            .action("llm.completion", "openai:gpt-4o")
            .subject(Subject { tenant: Some("acme-corp".into()), ..Default::default() }),
        |_ctx| async move {
            let summary = call_openai(document).await?;
            let actual = Amount::usd_microcents(estimate_actual(&summary));
            Ok((summary, actual))
        },
    ).await;

    match result {
        Ok(summary) => summary,
        Err(Error::BudgetExceeded { .. }) => {
            "Summary unavailable — budget limit reached.".into()
        }
        Err(e) => panic!("unexpected Cycles error: {e}"),
    }
}
```

:::

## The most common wrong integration

AI coding assistants frequently produce a plausible-but-wrong integration that *looks* like it's using Cycles, but enforces nothing. It records spend after the fact instead of authorizing it before the fact.

The blocks below show the structural difference using the real TypeScript SDK surface (`client.createReservation`, `client.commitReservation`). For most repos you should reach for `withCycles` from the language picker above — it does this lifecycle for you.

```typescript
// WRONG — calls OpenAI first, then reports usage to Cycles.
// This is observability, not enforcement. The model call has already
// happened and the money is already spent. DENY is meaningless here.
import { CyclesClient, CyclesConfig } from "runcycles";
const client = new CyclesClient(CyclesConfig.fromEnv());

const result = await openai.chat.completions.create({ /* ... */ });
await client.commitReservation("rsv_...", {
  actual: { amount: 35000, unit: "USD_MICROCENTS" },
});
```

```typescript
// RIGHT — reserve first, execute only on ALLOW, commit actuals after.
// On DENY the OpenAI call never fires.
import { CyclesClient, CyclesConfig } from "runcycles";
const client = new CyclesClient(CyclesConfig.fromEnv());

const reservation = await client.createReservation({
  idempotencyKey: crypto.randomUUID(),
  subject: { tenant: "acme-corp" },
  action: { kind: "llm.completion", name: "openai:gpt-4o" },
  estimate: { amount: 50000, unit: "USD_MICROCENTS" },
});
if (reservation.decision !== "ALLOW") {
  return fallback();
}
const result = await openai.chat.completions.create({ /* ... */ });
await client.commitReservation(reservation.reservationId, {
  actual: { amount: 35000, unit: "USD_MICROCENTS" },
});
```

The decorator / HOF / annotation in the language picker above does the reserve → check → execute → commit flow for you. Drop to the programmatic `CyclesClient` only when you need streaming, multi-step lifecycles, or a gateway integration. If a generated diff has the model call running before the Cycles primitive, it is wrong — reject it.

## Where to place Cycles in your architecture

The right insertion point depends on how the agent is structured. Hand the assistant this table along with the rest of the contract.

| Situation                       | Put Cycles here                                            |
| ------------------------------- | ---------------------------------------------------------- |
| Direct OpenAI / Anthropic call  | SDK wrapper around the model call                          |
| Tool-calling agent              | Tool execution wrapper, before the tool runs               |
| Multi-tenant SaaS agent         | Tenant / workflow boundary before each costly action       |
| MCP-based local assistant       | MCP for local discovery; runtime wrapper for production    |
| Gateway / proxy architecture    | Gateway, before the downstream model or tool call          |
| Batch / scheduled job           | Job entry point, around the per-item action                |

The constant: **the Cycles check is on the same code path as the costly action, and runs before it.** Anywhere else is logging, not enforcement.

## Success test: prove the downstream call is not made on DENY

This is the test the assistant must produce. It is the only thing that proves the integration is doing its job.

Mock Cycles so the guard returns DENY; do **not** mock the wrapped function itself. Replacing `generate_summary` / `generateSummary` bypasses the Cycles guard and only tests fallback handling.

::: code-group

```python [Python — pytest]
import importlib
from unittest.mock import MagicMock

def test_openai_not_called_when_budget_denied(monkeypatch, httpx_mock):
    monkeypatch.setenv("CYCLES_BASE_URL", "http://cycles.test")
    monkeypatch.setenv("CYCLES_API_KEY", "test-key")
    monkeypatch.setenv("CYCLES_TENANT", "acme-corp")
    httpx_mock.add_response(
        method="POST",
        url="http://cycles.test/v1/reservations",
        status_code=409,
        json={
            "error": "BUDGET_EXCEEDED",
            "message": "budget exhausted",
            "request_id": "req_test_123",
        },
    )

    import myapp.summary as summary
    summary = importlib.reload(summary)  # rebuild CyclesConfig.from_env() with test env
    fake_openai = MagicMock()
    monkeypatch.setattr(summary, "openai", fake_openai)

    result = summary.summarize_or_fallback("a document")

    assert "budget limit reached" in result
    fake_openai.chat.completions.create.assert_not_called()
```

```typescript [TypeScript — vitest]
import { afterEach, describe, it, expect, vi } from "vitest";

function mockCyclesDeny() {
  vi.stubGlobal(
    "fetch",
    vi.fn().mockResolvedValue({
      status: 409,
      statusText: "Conflict",
      json: () => Promise.resolve({
        error: "BUDGET_EXCEEDED",
        message: "budget exhausted",
        request_id: "req_test_123",
      }),
      headers: new Headers(),
    }),
  );
}

describe("summarizeOrFallback", () => {
  afterEach(() => {
    vi.resetModules();
    vi.unstubAllGlobals();
  });

  it("does not call OpenAI when budget is denied", async () => {
    process.env.CYCLES_BASE_URL = "http://cycles.test";
    process.env.CYCLES_API_KEY = "test-key";
    process.env.CYCLES_TENANT = "acme-corp";
    mockCyclesDeny();

    const create = vi.fn();
    vi.doMock("./openai-client", () => ({ openai: { chat: { completions: { create } } } }));

    const { summarizeOrFallback } = await import("./summary");
    const result = await summarizeOrFallback("a document");

    expect(result).toMatch(/budget limit reached/);
    expect(create).not.toHaveBeenCalled();
  });
});
```

```rust [Rust — test shape, adapt to your harness]
// Sketch only — adapt to your test harness. The Python and TypeScript tests
// above are runnable; this one outlines the equivalent structure.
//
// Use a mock CyclesClient that returns DENY (e.g. wiremock or a hand-rolled
// fake server). Inject a fake `call_openai` that increments a counter, and
// assert the counter is still zero after summarize_or_fallback returns the
// fallback string.
#[tokio::test]
async fn openai_not_called_when_budget_denied() {
    let calls = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
    let client = mock_cycles_client_returning_deny().await;

    let result = summarize_or_fallback_with_injected_caller(
        &client,
        "a document",
        {
            let calls = calls.clone();
            move |_doc| { calls.fetch_add(1, std::sync::atomic::Ordering::SeqCst); async { unreachable!() } }
        },
    ).await;

    assert!(result.contains("budget limit reached"));
    assert_eq!(calls.load(std::sync::atomic::Ordering::SeqCst), 0);
}
```

:::

If the assistant cannot make this test pass, the integration is wrong — the Cycles check is in the wrong place, or there is no deny path. Iterate until it passes.

## MCP availability is not enforcement

If your repo runs inside [Claude Desktop](/quickstart/mcp-claude-desktop), [Claude Code](/quickstart/mcp-claude-code), [Cursor](/quickstart/mcp-cursor), or [Windsurf](/quickstart/mcp-windsurf), registering the Cycles MCP server gives the host access to `cycles_reserve`, `cycles_commit`, `cycles_release`, and balance tools.

**MCP is useful for local assistant workflows and discovery. It is not, by itself, a hard runtime control unless the host or tool harness is required to call Cycles before executing the real action.** For production, the Cycles check must sit in the execution path — the SDK wrapper, gateway, or framework adapter — where the costly action cannot run without it. See [Integrating Cycles with MCP](/how-to/integrating-cycles-with-mcp) for the patterns that combine MCP discovery with hard enforcement.

## After the first wrap

Once the test passes:

1. Move to [shadow mode](/how-to/shadow-mode-in-cycles-how-to-roll-out-budget-enforcement-without-breaking-production) to observe decisions in production without blocking.
2. [Expand coverage](/how-to/adding-cycles-to-an-existing-application#stage-3-expand-coverage) to additional call paths.
3. Switch from shadow to live enforcement.
4. Wire the broader [reserve / commit / release lifecycle](/protocol/how-reserve-commit-works-in-cycles) — dynamic estimates, tenant scoping, run budgets.

The assistant has the contract; you have the test. Ship one boundary, prove it, expand.

## Protocol references (only if you need them)

Most integrations never touch the protocol directly — the decorator, HOF, and annotation hide it. Reach for these only when you need an exact field name, are debugging an error code, or are wrapping an HTTP call by hand because no SDK exists for your language. Linking these here so an AI coder can resolve the rare leak without guessing field names.

- [Reserve / commit lifecycle](/protocol/how-reserve-commit-works-in-cycles) — request/response shape for `reserve`, `commit`, `release`, plus `idempotencyKey`, `ttlMs`, and what each call returns.
- [Decide endpoint](/protocol/how-decide-works-in-cycles-preflight-budget-checks-without-reservation) — preflight check without holding budget. The right endpoint for shadow mode and "would this be allowed?" reads.
- [Error codes](/protocol/error-codes-and-error-handling-in-cycles) — canonical list (`BUDGET_EXCEEDED`, `RESERVATION_NOT_FOUND`, `INVALID_IDEMPOTENCY_KEY`, etc.). Map directly to `BudgetExceededError` / `CyclesProtocolException` in the SDKs.
- [Units (USD_MICROCENTS, TOKENS, CREDITS, RISK_POINTS)](/protocol/understanding-units-in-cycles-usd-microcents-tokens-credits-and-risk-points) — what `unit` and `amount` mean. The SDK examples on this page use `USD_MICROCENTS`; pick the right unit per ledger.
- [Caps and the three-way decision model](/protocol/caps-and-the-three-way-decision-model-in-cycles) — `ALLOW`, `ALLOW_WITH_CAPS`, `DENY`. If your wrapper handles only ALLOW/DENY, check this before assuming caps are safe to ignore.
- [Interactive OpenAPI reference](/api/) — full schema browser. The full spec is at `/cycles-protocol-v0.yaml`.

If you are an AI coding assistant: prefer the SDK-level integration on this page. Drop to protocol level only when the SDK genuinely doesn't expose what you need.



# Adding Cycles to an Existing Application

This guide covers how to incrementally add budget governance to an application that already makes LLM or API calls. Rather than rewriting your integration layer, you can adopt Cycles in stages.

::: tip MCP-compatible agents
If your agent runs in Claude Desktop, Claude Code, Cursor, or Windsurf, the fastest path is the [Cycles MCP Server](/quickstart/getting-started-with-the-mcp-server) — zero code changes needed. The guide below covers SDK-based integration for application code.
:::

::: tip Spring Boot / Java
This guide shows Python and TypeScript. For Spring Boot, equivalent patterns use the `@Cycles` annotation — see the [Spring Boot Quickstart](/quickstart/getting-started-with-the-cycles-spring-boot-starter) for full setup and examples.
:::

## The incremental adoption path

```
1. Shadow mode     →  Observe what enforcement would do, without blocking anything
2. Wrap one call   →  Add budget governance to a single LLM call path
3. Expand coverage →  Wrap additional call paths
4. Enforce         →  Switch from shadow mode to live enforcement
```

## Stage 1: Deploy Cycles and observe with the decide endpoint

Start by deploying the Cycles stack and using the **decide endpoint** as a side-channel to observe what budget decisions would be made without blocking any calls.

::: warning Important
The `dry_run` flag on the `@cycles` decorator / `withCycles` HOF skips executing the wrapped function entirely (it returns a `DryRunResult` instead). For shadow observation where your existing code still runs, use the `decide` endpoint separately.
:::

::: code-group
```python [Python]
import logging
import uuid

from runcycles import CyclesClient, CyclesConfig

logger = logging.getLogger(__name__)

config = CyclesConfig.from_env()
client = CyclesClient(config)

def existing_chat_function(prompt: str) -> str:
    # Observe: check what Cycles would decide, but don't block
    try:
        decision = client.decide({
            "idempotency_key": str(uuid.uuid4()),
            "subject": {"tenant": config.tenant},
            "action": {"kind": "llm.completion", "name": "openai:gpt-4o"},
            "estimate": {"amount": 2000000, "unit": "USD_MICROCENTS"},
        })
        logger.info("Cycles decision: %s", decision.get_body_attribute("decision"))
    except Exception:
        pass  # Don't let observation failures affect production

    # Your existing code — completely unchanged
    return call_openai(prompt)
```
```typescript [TypeScript]
import { CyclesClient, CyclesConfig } from "runcycles";

const client = new CyclesClient(CyclesConfig.fromEnv());

async function existingChatFunction(prompt: string) {
  // Observe: check what Cycles would decide, but don't block
  try {
    // The client posts wire-format (snake_case) bodies verbatim
    const decision = await client.decide({
      idempotency_key: crypto.randomUUID(),
      subject: { tenant: client.config.tenant! },
      action: { kind: "llm.completion", name: "openai:gpt-4o" },
      estimate: { amount: 2000000, unit: "USD_MICROCENTS" },
    });
    console.log("Cycles decision:", decision.body?.decision);
  } catch {
    // Don't let observation failures affect production
  }

  // Your existing code — completely unchanged
  return await callOpenAI(prompt);
}
```
:::

This approach lets you observe decisions in production logs while your existing code continues to run unmodified. Use the data to tune budgets before moving to enforcement.

See [Shadow Mode Rollout](/how-to/shadow-mode-in-cycles-how-to-roll-out-budget-enforcement-without-breaking-production) for the full guide on dry-run evaluation at the protocol level.

## Stage 2: Wrap your first call

Pick the highest-value call path to wrap first. Good candidates:

- The call that costs the most per invocation (e.g., GPT-4o or Claude Opus)
- The call that runs most frequently
- The call most likely to loop or retry

### Wrapping an existing function

**Before:**

::: code-group
```python [Python]
def generate_summary(document: str) -> str:
    response = openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": f"Summarize: {document}"}],
        max_tokens=2000,
    )
    return response.choices[0].message.content
```
```typescript [TypeScript]
async function generateSummary(document: string): Promise<string> {
  const response = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: `Summarize: ${document}` }],
    max_tokens: 2000,
  });
  return response.choices[0].message.content!;
}
```
:::

**After:**

::: code-group
```python [Python]
from runcycles import cycles

@cycles(
    estimate=lambda document: int((len(document) / 4 * 250 + 2000 * 1000) * 1.2),
    action_kind="llm.completion",
    action_name="openai:gpt-4o",
)
def generate_summary(document: str) -> str:
    response = openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": f"Summarize: {document}"}],
        max_tokens=2000,
    )
    return response.choices[0].message.content
```
```typescript [TypeScript]
import { withCycles } from "runcycles";

const generateSummary = withCycles(
  {
    estimate: (document: string) => Math.ceil((document.length / 4 * 250 + 2000 * 1000) * 1.2),
    actionKind: "llm.completion",
    actionName: "openai:gpt-4o",
  },
  async (document: string) => {
    const response = await openai.chat.completions.create({
      model: "gpt-4o",
      messages: [{ role: "user", content: `Summarize: ${document}` }],
      max_tokens: 2000,
    });
    return response.choices[0].message.content!;
  },
);
```
```java [Spring Boot]
import io.runcycles.client.java.spring.annotation.Cycles;

// The estimate is a raw SpEL expression — no #{...} template wrapper
@Cycles(value = "T(Math).ceil((#document.length() / 4 * 250 + 2000 * 1000) * 1.2)",
        actionKind = "llm.completion", actionName = "openai:gpt-4o")
public String generateSummary(String document) {
    // Same OpenAI call — business logic unchanged
    return openAiClient.chat(document);
}
```
:::

The only change is adding the `@cycles` decorator (Python), `withCycles` wrapper (TypeScript), or `@Cycles` annotation (Spring Boot). Your business logic stays exactly the same.

### Handling budget denial

Your existing error handling needs one new branch — what to do when budget is denied:

::: code-group
```python [Python]
from runcycles import BudgetExceededError

try:
    result = generate_summary(document)
except BudgetExceededError:
    # Option A: Return a graceful fallback
    result = "Summary unavailable — budget limit reached."
    # Option B: Use a cheaper model
    result = generate_summary_cheap(document)
    # Option C: Queue for later
    queue_for_retry(document)
```
```typescript [TypeScript]
import { BudgetExceededError } from "runcycles";

let result: string;

try {
  result = await generateSummary(document);
} catch (err) {
  if (!(err instanceof BudgetExceededError)) {
    throw err;
  }
  // Option A: Return a graceful fallback
  result = "Summary unavailable — budget limit reached.";
  // Option B: Use a cheaper model
  result = await generateSummaryCheap(document);
  // Option C: Queue for later
  queueForRetry(document);
}
```
```java [Spring Boot]
import io.runcycles.client.java.spring.model.CyclesProtocolException;

String result;

try {
    result = summaryService.generateSummary(document);
} catch (CyclesProtocolException e) {
    if (!e.isBudgetExceeded()) {
        throw e;
    }
    // Option A: Return a graceful fallback
    result = "Summary unavailable — budget limit reached.";
    // Option B: Use a cheaper model
    result = summaryService.generateSummaryCheap(document);
    // Option C: Queue for later
    queueForRetry(document);
}
```
:::

See [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) for a full treatment of fallback strategies.

## Stage 3: Expand coverage

Once the first call path is working, wrap additional calls. Use a consistent pattern:

::: code-group
```python [Python]
@cycles(estimate=500000, action_kind="llm.completion", action_name="openai:gpt-4o-mini")
def classify_intent(text: str) -> str:
    ...

@cycles(estimate=3000000, action_kind="llm.completion", action_name="openai:gpt-4o")
def generate_response(context: str, intent: str) -> str:
    ...

@cycles(estimate=100000, action_kind="tool.call", action_name="web-search")
def search_web(query: str) -> list:
    ...
```
```typescript [TypeScript]
const classifyIntent = withCycles(
  { estimate: 500000, actionKind: "llm.completion", actionName: "openai:gpt-4o-mini" },
  async (text: string) => { ... },
);

const generateResponse = withCycles(
  { estimate: 3000000, actionKind: "llm.completion", actionName: "openai:gpt-4o" },
  async (context: string, intent: string) => { ... },
);

const searchWeb = withCycles(
  { estimate: 100000, actionKind: "tool.call", actionName: "web-search" },
  async (query: string) => { ... },
);
```
:::

Each wrapped function reserves independently. If the agent calls all three in sequence, the total budget consumed is the sum of actual usage — and each call is individually authorized before it runs.

## Stage 4: Switch to live enforcement

Wrapped calls enforce immediately unless `dry_run` is set — so if you followed Stages 1–3 as written, your wrapped paths are already live, and Stage 4 simply means retiring the Stage 1 `decide` side-channel once you're confident in your budget allocations. If you did set `dry_run=True` on any wrapped call during testing (recall from Stage 1 that it skips executing the wrapped function, so it's unsuitable for shadow-observing production traffic), remove it now:

::: code-group
```python [Python]
# Remove dry_run to enable enforcement
@cycles(
    estimate=2000000,
    action_kind="llm.completion",
    action_name="openai:gpt-4o",
    # dry_run=True,  ← remove this line
)
def generate_summary(document: str) -> str:
    ...
```
```typescript [TypeScript]
// Remove dryRun to enable enforcement
const generateSummary = withCycles(
  {
    estimate: 2000000,
    actionKind: "llm.completion",
    actionName: "openai:gpt-4o",
    // dryRun: true,  ← remove this line
  },
  async (document: string) => { ... },
);
```
```java [Spring Boot]
// Remove dryRun to enable enforcement
@Cycles(value = "2000000",
        actionKind = "llm.completion", actionName = "openai:gpt-4o"
        // dryRun = true  ← remove this line
)
public String generateSummary(String document) { ... }
```
:::

## Tips for existing applications

### Keep your existing error handling

Don't replace your existing try/except or try/catch blocks. Add Cycles error handling alongside them:

```python
try:
    result = generate_summary(document)
except BudgetExceededError:
    result = "Budget limit reached."
except openai.APIError as e:
    # Your existing error handling stays
    result = handle_openai_error(e)
```

### Start with generous budgets

When first deploying, set budgets higher than you think you need. You can tighten them after collecting real usage data. Under-budgeting on day one creates unnecessary friction.

### Use scopes to separate environments

Use different workspace scopes for dev, staging, and production:

```python
@cycles(
    estimate=2000000,
    action_kind="llm.completion",
    action_name="openai:gpt-4o",
    workspace=os.environ.get("ENVIRONMENT", "dev"),  # dev, staging, prod
)
def generate_summary(document: str) -> str:
    ...
```

### Don't wrap everything at once

It's better to have 3 well-instrumented call paths than 30 poorly-estimated ones. Start with the calls that matter most.

## Next steps

- [Shadow Mode Rollout](/how-to/shadow-mode-in-cycles-how-to-roll-out-budget-enforcement-without-breaking-production) — full guide to safe rollout
- [Cost Estimation Cheat Sheet](/how-to/cost-estimation-cheat-sheet) — how much to reserve per model
- [Choosing the Right Integration Pattern](/how-to/choosing-the-right-integration-pattern) — decorator vs programmatic vs middleware
- [Integrations Overview](/how-to/integrations-overview) — all supported frameworks and providers
- [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) — what to do when budget is denied



# API Key Management in Cycles

Every request to the Cycles server requires an API key. This page explains how API keys work, how to create and manage them, and how they relate to tenant isolation.

## How API keys work

The Cycles server authenticates requests using the `X-Cycles-API-Key` header. Each API key is associated with exactly one tenant.

When a request arrives:

1. The server extracts the `X-Cycles-API-Key` header
2. Validates the key exists and is active
3. Derives the effective tenant from the key
4. Verifies that `subject.tenant` in the request body matches the key's tenant
5. If any check fails, returns `401 UNAUTHORIZED` or `403 FORBIDDEN`

This ensures strict tenant isolation — an API key for tenant A cannot create reservations or query balances for tenant B.

## Key states

An API key can be in one of three states:

| State | Meaning |
|---|---|
| `ACTIVE` | Key is valid and can be used for requests |
| `REVOKED` | Key has been manually disabled |
| `EXPIRED` | Key has passed its expiration date |

Only `ACTIVE` keys are accepted. Requests with `REVOKED` or `EXPIRED` keys receive `401 UNAUTHORIZED`.

## Creating API keys

API keys are managed through the [Cycles Admin](https://github.com/runcycles/cycles-server-admin) interface:

```bash
# Create a new API key for a tenant
curl -X POST http://localhost:7979/v1/admin/api-keys \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{
    "tenant_id": "acme",
    "name": "production-chatbot",
    "description": "Production chatbot key",
    "permissions": ["reservations:create", "reservations:commit", "reservations:release", "balances:read"]
  }'
```

<a id="available-permissions"></a>
### Available permissions (27 total)

**Tenant-scoped permissions** (used with `X-Cycles-API-Key` for runtime and tenant operations):

| Permission | Grants | Default |
|---|---|---|
| `reservations:create` | Create new reservations | Yes |
| `reservations:commit` | Commit existing reservations | Yes |
| `reservations:release` | Release existing reservations | Yes |
| `reservations:extend` | Extend reservation TTL | Yes |
| `reservations:list` | List reservations | Yes |
| `balances:read` | Query balance information | Yes |
| `webhooks:write` | Create, update, delete, and test tenant webhooks at `/v1/webhooks` | No |
| `webhooks:read` | List tenant webhooks and delivery history | No |
| `events:read` | Query tenant event stream at `/v1/events` | No |

**Tenant budget/policy permissions** (v0.1.25.6+):

| Permission | Grants | Default? |
|---|---|---|
| `budgets:read` | List and read own tenant budgets | Yes |
| `budgets:write` | Create and fund own tenant budgets | Yes |
| `policies:read` | List and read own tenant policies | Yes |
| `policies:write` | Create and update own tenant policies | Yes |

**Admin wildcard permissions** (used with `X-Cycles-API-Key` for admin operations):

| Permission | Grants |
|---|---|
| `admin:read` | Satisfies **any** `*:read` permission (budgets:read, policies:read, webhooks:read, events:read, etc.) |
| `admin:write` | Satisfies **any** `*:write` permission (budgets:write, policies:write, webhooks:write, etc.). Does NOT grant read access — use both `admin:read` and `admin:write` for full access. |

::: tip Wildcard behavior (v0.1.25.7+)
`admin:write` acts as a server-level wildcard — it satisfies any `*:write` permission requirement. This means pre-v0.1.25.6 keys with `admin:write` continue to work without migration even after granular permissions were introduced. `admin:read` does NOT satisfy `*:write`.
:::

**Admin granular permissions** (v0.1.25+ — finer-grained alternative to admin wildcards):

| Permission | Grants |
|---|---|
| `admin:tenants:read` | Read tenant details |
| `admin:tenants:write` | Create and update tenants |
| `admin:budgets:read` | List and read budgets |
| `admin:budgets:write` | Create, fund, and update budgets |
| `admin:policies:read` | List and read policies |
| `admin:policies:write` | Create and update policies |
| `admin:apikeys:read` | List API keys |
| `admin:apikeys:write` | Create and revoke API keys |
| `admin:webhooks:read` | List admin webhook subscriptions and deliveries |
| `admin:webhooks:write` | Create, update, delete, test, and replay admin webhooks |
| `admin:events:read` | Query admin event stream |
| `admin:audit:read` | Query audit logs |

> **Defaults:** When no permissions are specified at key creation, the key receives 10 default permissions: the 6 runtime permissions (`reservations:create`, `reservations:commit`, `reservations:release`, `reservations:extend`, `reservations:list`, `balances:read`) plus `budgets:read`, `budgets:write`, `policies:read`, `policies:write`. Webhook, event, and admin permissions must be explicitly requested.

A typical runtime key needs only the 6 runtime permissions (`reservations:*` and `balances:read`) — remember that an explicit `permissions` array replaces the 10-value default set, so include `budgets:read`/`budgets:write` if the key also manages budgets. Add `webhooks:write` and `webhooks:read` for [webhook subscriptions](/how-to/managing-webhooks#tenant-self-service). Cross-tenant admin operations require the admin server's `X-Admin-API-Key` — no tenant-key permission (including the legacy `admin:read`/`admin:write` wildcards) grants access to those endpoints.

::: warning Admin permissions on tenant keys (v0.1.25.7)
`admin:read` and `admin:write` are accepted on tenant keys for backward compatibility, but **SHOULD NOT be assigned to new tenant keys**. Use the specific permissions (`budgets:write`, `policies:read`, etc.) instead. The admin key (`X-Admin-API-Key`) is server-configured and is not provisioned through the API key creation endpoint.

No permission on a tenant key — including `admin:read`/`admin:write` and the granular `admin:*` permissions — ever grants access to **AdminKeyAuth-only** endpoints (tenant management, API key management, audit, admin webhooks/events/config/overview). Those endpoints are unreachable with a tenant key; they accept only the server-configured `X-Admin-API-Key` header.
:::

For the full endpoint-to-header-to-permission mapping, see the [Architecture Overview — Authentication](/quickstart/architecture-overview-how-cycles-fits-together#authentication).

Response:

```json
{
  "key_id": "key_abc123...",
  "key_secret": "cyc_live_abc123...",
  "key_prefix": "cyc_live_abc12",
  "tenant_id": "acme",
  "permissions": ["reservations:create", "reservations:commit", "reservations:release", "balances:read"],
  "created_at": "2026-03-01T00:00:00Z",
  "expires_at": "2026-05-30T00:00:00Z"
}
```

Store the API key securely. It is shown only once at creation time.

## Using API keys

### In the Python client

Configure the key via `CyclesConfig`:

```python
import os
from runcycles import CyclesConfig

config = CyclesConfig(
    base_url="http://localhost:7878",
    api_key=os.environ["CYCLES_API_KEY"],
    tenant="acme",
)
```

Or from environment variables:

```bash
export CYCLES_BASE_URL=http://localhost:7878
export CYCLES_API_KEY=cyc_live_abc123...
export CYCLES_TENANT=acme
```

```python
config = CyclesConfig.from_env()
```

### In the Spring Boot Starter

Configure the key in your project's `application.yml`:

```yaml
cycles:
  api-key: ${CYCLES_API_KEY}
  base-url: http://localhost:7878
  tenant: acme
```

Use an environment variable rather than hardcoding the key.

### In direct HTTP calls

Pass the key in the `X-Cycles-API-Key` header:

```bash
curl -X POST http://localhost:7878/v1/reservations \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: cyc_live_abc123..." \
  -d '{ ... }'
```

## Listing API keys

`GET /v1/admin/api-keys` lists keys. The endpoint is **AdminKeyAuth-only** — it requires the `X-Admin-API-Key` header, and tenant keys (`X-Cycles-API-Key`) cannot call it regardless of the permissions they carry. `tenant_id` is an optional filter: omit it to list keys across all tenants (v0.1.25.22+), or provide it to scope the result to a single tenant. Cross-tenant results are cursor-paginated; the cursor format is an implementation detail — treat it as an opaque string and pass it back unchanged.

```bash
# Cross-tenant — all keys, newest first
curl -G "http://localhost:7979/v1/admin/api-keys" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  --data-urlencode "sort_by=created_at" \
  --data-urlencode "sort_dir=desc" \
  --data-urlencode "limit=50" | jq .

# Tenant-scoped
curl -G "http://localhost:7979/v1/admin/api-keys?tenant_id=acme" \
  -H "X-Admin-API-Key: $ADMIN_KEY" | jq .
```

`search` (v0.1.25.25+) does a case-insensitive match over `key_id` and `name`. See [Searching and Sorting Admin List Endpoints](/how-to/searching-and-sorting-admin-list-endpoints) for the full parameter vocabulary.

## Revoking API keys

::: tip Revoke from the dashboard
Key revocation is also a one-click action on the API Keys page in the [Cycles Admin Dashboard](/quickstart/deploying-the-cycles-dashboard) — typically faster than crafting a curl when responding to a leaked-key incident. Revocation is irreversible; the dashboard surfaces a confirmation step.
:::

Revoke a key to immediately block all requests using it:

```bash
curl -X DELETE "http://localhost:7979/v1/admin/api-keys/key_abc123?reason=leaked+in+ci+logs" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY"
```

The optional `reason` query parameter is recorded for the audit trail. Revoking a key that is already revoked returns `409` (`ALREADY_REVOKED`).

Revocation is immediate. Any in-flight requests using the revoked key will fail on their next call to the Cycles server. Active reservations created with the revoked key remain valid until they expire or are committed/released.

::: tip Revocation, not deletion
The `DELETE` endpoint performs a **status transition** (ACTIVE → REVOKED), not a hard delete. The key record is retained so that audit logs referencing the key remain resolvable. This is consistent with the lifecycle model used across Cycles — see the equivalent notes on [tenant closure](/how-to/tenant-creation-and-management-in-cycles#closed) and [budget decommissioning](/how-to/budget-allocation-and-management-in-cycles#resizing-a-budget-reset).
:::

## Updating a key without rotating

`PATCH /v1/admin/api-keys/{key_id}` (AdminKeyAuth-only) performs a partial update — only the fields you send are modified. Mutable fields: `permissions` (replaces the full set), `scope_filter`, and `name`/`description`/`metadata`. `tenant_id`, `key_id`, `key_prefix`, `expires_at`, and `status` are immutable — to change expiry or tenant, revoke and recreate. Patching a revoked or expired key returns `409`.

## Key rotation

To rotate an API key without downtime:

1. Create a new key for the same tenant
2. Update your application configuration to use the new key
3. Deploy the configuration change
4. Verify traffic is flowing with the new key
5. Revoke the old key

Because both keys are valid during the transition, there is no interruption.

## Tenant isolation

API keys are the primary mechanism for tenant isolation in Cycles.

**Enforced behaviors:**

- A key for tenant A can only create reservations where `subject.tenant = "A"` (or where tenant is omitted, in which case the server uses the key's tenant)
- A key for tenant A cannot commit, release, or extend reservations owned by tenant B
- A key for tenant A cannot query balances for tenant B
- A key for tenant A cannot list reservations belonging to tenant B

**If the tenant is omitted from the Subject**, the server automatically sets it to the key's associated tenant. This is the recommended approach — configure the tenant at the key level and let the server enforce it.

## Best practices

### One key per environment

Use separate API keys for production, staging, and development, even within the same tenant. This makes it easy to revoke a single environment's access without affecting others.

### Use environment variables

Never hardcode API keys in source code:

```yaml
# Good
cycles:
  api-key: ${CYCLES_API_KEY}

# Bad
cycles:
  api-key: cyc_live_abc123...
```

### Minimal key scope

If you operate multiple tenants, issue one key per tenant. Do not share keys across tenants.

### Monitor key usage

Track which keys are making requests. If a key is compromised, revoke it immediately and issue a replacement.

## Error responses

| Error | HTTP | When |
|---|---|---|
| `UNAUTHORIZED` | 401 | Missing `X-Cycles-API-Key` header, or key is invalid/revoked/expired |
| `FORBIDDEN` | 403 | Key is valid but `subject.tenant` does not match the key's tenant |
| `INSUFFICIENT_PERMISSIONS` | 403 | Key is valid but lacks the required permission for the endpoint (e.g., calling `POST /v1/admin/budgets/fund` without `budgets:write`) |

## Next steps

- [Tenant Creation and Management](/how-to/tenant-creation-and-management-in-cycles) — create and manage the tenants that API keys belong to
- [Authentication and Tenancy](/protocol/authentication-tenancy-and-api-keys-in-cycles) — deeper dive into the auth model
- [Self-Hosting the Cycles Server](/quickstart/self-hosting-the-cycles-server) — deploy your own instance
- [Architecture Overview](/quickstart/architecture-overview-how-cycles-fits-together) — how authentication fits into the system



# How to Assign RISK_POINTS to AI Agent Tools

This guide walks you through an illustrative method for assigning [RISK_POINTS](/glossary#risk-points) to tool attempts. Cycles treats those caller-assigned points as a budget unit; the host still authenticates the principal, authorizes the tool and arguments, and makes the reservation boundary mandatory.

For the full risk assessment framework and regulatory context, see [AI Agent Risk Assessment](/blog/ai-agent-risk-assessment-score-classify-enforce-tool-risk). This page is the implementation-focused quick reference.

**Prerequisites:**
- You know what RISK_POINTS are ([protocol reference](/protocol/understanding-units-in-cycles-usd-microcents-tokens-credits-and-risk-points))
- You have a list of tools your agent can call

## Step 1: List your tools

Start by inventorying every tool the agent can invoke. For each tool, note:

- **Name** — the function or tool name your agent calls
- **What it does** — one-sentence description
- **What it modifies** — does it read, write locally, call external APIs, mutate records, or execute irreversible actions?

Example inventory for a customer support agent:

| Tool | What it does | What it modifies |
|---|---|---|
| `search_knowledge` | Searches the knowledge base | Nothing (read-only) |
| `get_customer` | Looks up customer by ID | Nothing (read-only) |
| `save_draft_note` | Saves internal draft note | Local, reversible |
| `call_crm_api` | Reads customer data from CRM | External API (read) |
| `send_customer_email` | Sends email to customer's inbox | External, hard to reverse |
| `update_crm_status` | Updates customer status in CRM | External, hard to reverse |
| `create_jira_ticket` | Creates a Jira ticket (triggers notifications) | External, customer-visible |
| `issue_refund` | Processes a financial refund | Irreversible |

## Step 2: Classify each tool into a risk tier

Use these three questions to classify:

```
Does this tool modify any state?
├── NO → Tier 0 (Read-only)
│
└── YES
    ├── Can a human undo this in under 5 minutes?
    │   └── YES → Tier 1 (Write-local)
    │
    ├── Does it leave your system boundary (external API)?
    │   └── YES, but reversible → Tier 2 (Write-external)
    │
    ├── Does it affect someone who did not request it?
    │   (customer, external party, end user)
    │   └── YES, hard to reverse → Tier 3 (Mutation)
    │
    └── Is the change irreversible? (deploy, payment, permission grant)
        └── YES → Tier 4 (Execution)
```

The third question — **"Does it affect someone who did not request it?"** — is the key differentiator between Tier 2 (external but contained) and Tier 3 (customer-facing impact). An API call to a third-party service you control is Tier 2. An email landing in a customer's inbox is Tier 3 — even though both are external.

This guide proposes the following **starting scores**. They are not built into the server and are not a compliance standard:

| Tier | Type | Base Points | Rationale |
|:---:|---|:---:|---|
| 0 | Read-only | 0 | No side effects — reads should be free |
| 1 | Write-local | 1 | Low impact, easily reversible |
| 2 | Write-external | 5 | External dependency, some coordination to reverse |
| 3 | Mutation | 20 | Customer-facing impact, difficult to reverse |
| 4 | Execution | 50 | Irreversible, financial or production impact |

::: tip Edge cases and rules of thumb
- **If a tool matches multiple tiers, assign the highest.** A generic API tool that can both read and write should be scored at the write tier.
- **If a tool's risk varies by parameter** (e.g., a generic `call_api` used for both reads and writes), score at the highest tier it can reach, or split it into separate tool definitions.
- **If unsure, score one tier higher and validate in shadow mode.** It's cheaper to loosen a tight score than to recover from an under-scored tool causing an incident.
- **When multiplied scores produce decimals** (e.g., 5 × 1.5 = 7.5), round up to the next integer.
- The same tool in different deployments can have different tiers based on what it connects to.
:::

## Step 3: Apply risk scoring multipliers

Base points assume a generic context. Four factors adjust them for your specific deployment. **Take the maximum** of all four — they don't stack multiplicatively.

**Formula:** `Final RISK_POINTS = Base × max(Audience, Sensitivity, Regulatory, Reputational)`

### Audience size

| Audience | Multiplier |
|---|:---:|
| Internal only | 1x |
| Single external party | 1.5x |
| Customer segment (<100) | 2x |
| Broad external (100+) | 3x |

### Data sensitivity

| Sensitivity | Multiplier |
|---|:---:|
| Public data | 1x |
| Internal business data | 1.5x |
| Customer PII | 2x |
| Financial / health / regulated | 3x |

### Regulatory context

| Regulatory | Multiplier |
|---|:---:|
| No specific regulation | 1x |
| General data protection (e.g., GDPR) | 1.5x |
| Industry-specific (e.g., HIPAA, PCI-DSS, SOX) | 2x |
| Multiple overlapping regulations | 3x |

### Reputational exposure

| Exposure | Multiplier |
|---|:---:|
| Internal only | 1x |
| Single external party | 1.5x |
| Customer-facing (visible to end users) | 2x |
| Press / social media / regulatory scrutiny | 3x |

### Applying the formula

For `send_customer_email`:
- Base: Tier 3 = 20 points
- Audience: customer segment (2x)
- Sensitivity: internal business data (1.5x)
- Regulatory: none (1x)
- Reputational: customer-facing (2x)
- Max multiplier: **2x** (audience or reputational)
- Final: 20 × 2 = **40 RISK_POINTS**

## Step 4: Set your per-run budget

Sum the expected tool calls for representative runs, then choose a buffer from the variation you are prepared to allow:

1. Count how many times each tool is called in a normal agent run
2. Multiply each by its RISK_POINTS score
3. Sum the results
4. Choose a buffer from observed percentiles and the maximum tolerated exposure

**Example: Customer support agent**

| Tool | Score | Typical calls per run | Points consumed |
|---|:---:|:---:|:---:|
| `search_knowledge` | 0 | 4-6 | 0 |
| `get_customer` | 0 | 1-2 | 0 |
| `save_draft_note` | 1 | 1 | 1 |
| `call_crm_api` | 5 | 1 | 5 |
| `send_customer_email` | 40 | 1 | 40 |
| `update_crm_status` | 40 | 1 | 40 |
| `create_jira_ticket` | 20 | 0-1 | 0-20 |
| `issue_refund` | 150 | 0 (rare) | 0 |
| **Normal run total** | | | **86-106** |

Set per-run budget to **250 RISK_POINTS**:
- Normal resolution: ~86-106 points (comfortable headroom)
- Complex resolution (email + ticket + CRM update): ~106 points
- Single refund + email: 150 + 40 = 190 points (fits)
- Two refunds: 300 points (does **not** fit — requires escalation)

At a mandatory boundary, this illustrative budget would reject an attempt whose submitted points no longer fit. It does not itself determine whether a refund or email is authorized.

To enforce this per run, create the ledger at a unique workflow scope such as `workflow:run-12345` and send that run ID as `subjects.workflow` on every protected tool attempt. `run` is not a native scope, and `dimensions.run_id` is attribution only.

## Step 5: Validate with shadow mode

Before enforcing, run with [`dry_run: true`](/protocol/dry-run-shadow-mode-evaluation-in-cycles) long enough to cover representative traffic:

1. Create the RISK_POINTS budget via admin API
2. Set reservations to dry-run mode
3. Have the application log every dry-run result and actual outcome, then review the hypothetical denial rate by workflow and action class. The current server emits `reservation.denied` for denied evaluations, but that is not a complete shadow dataset.
   - A high rate can indicate scores or budgets are too tight, incomplete scoping, or genuinely abnormal traffic.
   - A low rate does not prove that the policy is effective; exercise known denial cases separately.
4. Adjust individual tool scores or the run budget based on the data
5. When denial patterns are no longer surprising, enable enforcement

See the [shadow mode rollout guide](/how-to/shadow-mode-in-cycles-how-to-roll-out-budget-enforcement-without-breaking-production) for the full process.

## Common AI agent tool classifications

Reference table for 25+ common tools. **These are starting points** — adjust based on your deployment context using the multiplier framework above.

### Tier 0 — Read-only (0 base points)

| Tool | Description | Typical final score |
|---|---|:---:|
| `search` / `search_knowledge` | Search knowledge base, vector DB | 0 |
| `get_user` / `get_customer` | Look up user/customer record | 0 |
| `read_file` | Read a local file | 0 |
| `get_weather` | Weather API lookup | 0 |
| `web_search` | Web search (Google, Bing) | 0 |
| `list_records` | List/query database records | 0 |
| `get_balance` | Check account balance | 0 |

::: tip Reads in regulated environments
If your agent handles PII, financial, or health data, consider assigning 1-2 points to sensitive read tools or tracking them under a separate budget. An agent that reads 10,000 customer records hasn't changed state, but has created a data-exposure surface.
:::

### Tier 1 — Write-local (1 base point)

| Tool | Description | Typical final score |
|---|---|:---:|
| `save_draft` | Save draft document | 1 |
| `log_event` | Write to application log | 1 |
| `update_cache` | Update local cache | 1 |
| `save_note` / `add_internal_note` | Save internal note | 1-2 |

### Tier 2 — Write-external (5 base points)

| Tool | Description | Typical final score |
|---|---|:---:|
| `call_external_api` | Generic third-party API call | 5-10 |
| `webhook_post` | Fire a webhook | 5-10 |
| `post_to_internal_slack` | Post to an internal Slack channel | 5 |
| `generate_image` | Generate image via API (Stable Diffusion, DALL-E) | 5-10 |
| `generate_video` | Generate video via API | 10-15 |

### Tier 3 — Mutation (20 base points)

| Tool | Description | Typical final score |
|---|---|:---:|
| `send_email` / `send_customer_email` | Send email to customer | 40-60 |
| `create_jira_ticket` / `create_support_ticket` | Create ticket (triggers customer-visible notifications) | 20-40 |
| `send_external_slack` | Post to customer-shared Slack channel | 40-60 |
| `update_record` / `update_crm` | Update database/CRM record | 20-40 |
| `delete_record` | Delete a database record | 40-60 |
| `create_calendar_event` | Schedule meeting on someone's calendar | 20-40 |
| `update_subscription` | Modify a customer's subscription | 40-60 |

### Tier 4 — Execution (50 base points)

| Tool | Description | Typical final score |
|---|---|:---:|
| `deploy` / `trigger_deploy` | Deploy to production | 100-150 |
| `execute_payment` / `issue_refund` | Process financial transaction | 100-150 |
| `grant_permission` | Modify access control | 100-150 |
| `execute_code` | Run arbitrary code | 100 |
| `delete_database` | Drop a table or database | 150 |
| `modify_infrastructure` | Change cloud infrastructure (scaling, networking) | 100-150 |

## Worked example: Risk scoring an AI agent from scratch

A data analysis agent with these tools:

1. **`query_database`** — reads data → Tier 0, 0 points
2. **`web_search`** — external search → Tier 0, 0 points (read-only)
3. **`generate_chart`** — calls chart API → Tier 2, 5 points (external, reversible)
4. **`send_report_email`** — emails stakeholders → Tier 3, 20 × 2x (customer segment audience) = **40 points**
5. **`update_dashboard`** — writes to internal dashboard → Tier 1, 1 point (easily reversible)
6. **`export_to_s3`** — writes file to S3 bucket → Tier 2, 5 × 1.5x (internal business data) ≈ **8 points** (round up)

Typical run: 10 queries (0) + 3 searches (0) + 2 charts (10) + 1 email (40) + 1 dashboard update (1) + 1 export (8) = **59 points**

Set run budget: **100 RISK_POINTS** — at a mandatory boundary, the third 40-point report attempt would not fit after two such attempts, assuming no other point consumption.

## When to recalibrate risk scores

Review your scores when:

- **A new tool is added.** Classify and score it before deployment.
- **An incident occurs.** If a tool caused damage, re-evaluate its tier and multiplier.
- **Usage patterns shift.** If shadow mode shows agents consistently near the budget ceiling on normal runs, the budget is too tight — raise it or optimize the workflow.
- **The consequence model changes.** Revisit scores when a tool gains new destinations, broader permissions, more sensitive data, or a larger audience.

## Next steps

- [Budget Allocation and Management](/how-to/budget-allocation-and-management-in-cycles) — configure your scored tool list and budget via the admin API
- [Common Budget Patterns](/how-to/common-budget-patterns) — per-tenant, per-workflow, per-run budget structures
- [Shadow Mode Rollout](/how-to/shadow-mode-in-cycles-how-to-roll-out-budget-enforcement-without-breaking-production) — validating scores before enforcing
- [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) — what agents should do when risk budget runs low
- [AI Agent Risk Assessment](/blog/ai-agent-risk-assessment-score-classify-enforce-tool-risk) — the full framework deep-dive with regulatory context
- [Event Payloads Reference](/protocol/event-payloads-reference) — monitoring commit overage and denial events



# Budget Allocation and Management in Cycles

Before you can enforce budgets with Cycles, budgets need to be allocated to scopes. This page explains how budget allocation works, how to set it up, and how to manage budgets over time.

## What is allocation?

Allocation is the total budget assigned to a scope. It is the ceiling against which reservations and commits are measured.

The formula for remaining budget is:

```
remaining = allocated - spent - reserved - debt
```

A reservation succeeds only if `remaining >= estimate` across all affected scopes.

## How allocation works

Each scope in Cycles has an `allocated` value. When a client creates a reservation, the server checks the allocated budget for every scope in the derived hierarchy.

For example, if a reservation targets:

```json
{
  "tenant": "acme",
  "workspace": "production",
  "app": "chatbot"
}
```

Three scopes are checked:

- `tenant:acme` — must have sufficient remaining budget
- `tenant:acme/workspace:production` — must have sufficient remaining budget
- `tenant:acme/workspace:production/app:chatbot` — must have sufficient remaining budget

All three must pass for the reservation to succeed.

## Setting budgets

Budget allocation is managed through the [Cycles Admin Server](https://github.com/runcycles/cycles-server-admin) API (port 7979 by default). The admin server and the runtime Cycles server share the same Redis instance.

### Authentication

Budget, policy, and balance endpoints on the admin server require a tenant-scoped API key (`X-Cycles-API-Key`) with the appropriate permissions:

- **`budgets:write`** — required for creating budgets, funding, and resetting (or `admin:write` as wildcard)
- **`budgets:read`** — required for listing and querying budgets (or `admin:read` as wildcard)
- **`policies:write`** — required for creating and updating policies (or `admin:write` as wildcard)
- **`policies:read`** — required for listing and querying policies (or `admin:read` as wildcard)

Default API keys (created without explicit permissions) include `budgets:write` and `budgets:read` as of v0.1.25.6 and will work for budget operations. Keys created before v0.1.25.6 with explicitly specified permission sets may need `budgets:write` and/or `budgets:read` added. See [API Key Management](/how-to/api-key-management-in-cycles#available-permissions) for the full permission list.

::: warning X-Admin-API-Key vs X-Cycles-API-Key
The bootstrap admin key (`X-Admin-API-Key`) is used for tenant management, API key management, audit log access, and budget PATCH/freeze/unfreeze (admin-only operations). Budget **create**, **list**, and **fund** are dual-auth — they accept either `X-Cycles-API-Key` (with `budgets:write`/`budgets:read`) or `X-Admin-API-Key`. Under the admin key: create requires `tenant_id` in the request **body** (and it must be omitted under a tenant key — the tenant is implicit); fund requires the `tenant_id` **query parameter**; list treats `tenant_id` as an optional filter (omit it for a cross-tenant listing). Admin-key writes are audit-logged with `actor_type=admin_on_behalf_of` (the lowercase wire value; spec prose sometimes shows the enum constant `ADMIN_ON_BEHALF_OF`).
:::

### Using the Cycles Admin API

Create budget ledgers and fund them via the admin API:

```bash
# Create a tenant budget ledger
curl -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "scope": "tenant:acme",
    "unit": "USD_MICROCENTS",
    "allocated": { "amount": 1000000, "unit": "USD_MICROCENTS" }
  }'

# Fund the budget
curl -X POST "http://localhost:7979/v1/admin/budgets/fund?scope=tenant:acme&unit=USD_MICROCENTS" \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "operation": "CREDIT",
    "amount": { "amount": 1000000, "unit": "USD_MICROCENTS" },
    "idempotency_key": "fund-acme-001"
  }'

# Create a workspace budget within that tenant
curl -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "scope": "tenant:acme/workspace:production",
    "unit": "USD_MICROCENTS",
    "allocated": { "amount": 500000, "unit": "USD_MICROCENTS" }
  }'

curl -X POST "http://localhost:7979/v1/admin/budgets/fund?scope=tenant:acme/workspace:production&unit=USD_MICROCENTS" \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "operation": "CREDIT",
    "amount": { "amount": 500000, "unit": "USD_MICROCENTS" },
    "idempotency_key": "fund-acme-prod-001"
  }'
```

::: info Note
Tenants and API keys must be created first using the admin key (`X-Admin-API-Key`). See [Deploying the Full Cycles Stack](/quickstart/deploying-the-full-cycles-stack) for the complete bootstrap sequence.
:::

### Budget hierarchy

Budgets are independent at each scope level. A tenant budget of 1,000,000 does not automatically distribute to child scopes.

You set the allocated amount at each level you want to control:

| Scope | Allocated | Purpose |
|---|---|---|
| `tenant:acme` | 1,000,000 | Global cap for the tenant |
| `tenant:acme/workspace:production` | 500,000 | Cap for the production environment |
| `tenant:acme/workspace:production/app:chatbot` | 100,000 | Cap for the chatbot app |

A reservation for 10,000 against the chatbot scope must pass all three levels.

### Unallocated scopes

If a scope has a budget ledger with zero allocation (`allocated = 0`), any reservation targeting it will be denied with `BUDGET_EXCEEDED` (409). The ledger exists but has no room.

If a scope has no budget ledger at all, it is **skipped** during enforcement — it does not block the reservation. This is different from zero allocation: a missing ledger is ignored, a zero-allocation ledger is enforced.

### How budget lookup works during reservations

When the server processes a reservation, it derives scope paths from the subject (e.g., `tenant:acme`, `tenant:acme/workspace:prod`, `tenant:acme/workspace:prod/app:chatbot`) and checks each for a budget ledger:

1. Scopes **with** a budget ledger are checked for sufficient funds
2. Scopes **without** a budget ledger are skipped — they do not block the reservation
3. If **no** derived scope has a budget ledger, the reservation is rejected with `NOT_FOUND` (404) — the response message is `"Budget not found for provided scope: ..."`. (On `/v1/decide` and dry-run reserve, the same condition surfaces as `200 DENY` with `reason_code=BUDGET_NOT_FOUND`.)
4. If **any** budgeted scope has insufficient funds, the reservation is rejected with `BUDGET_EXCEEDED` (409)

This means you only need budgets at the scope levels where you want enforcement. For example, if you only set a tenant-level budget, workspace and app scopes are skipped — the tenant budget is the only constraint.

| Scenario | Result |
|---|---|
| Budget at tenant only, reservation targets tenant/workspace/app | Reserves against tenant budget; workspace and app skipped |
| Budget at tenant and app, not workspace | Reserves against both; workspace skipped |
| No budget at any scope | `NOT_FOUND` (404) — message: `"Budget not found for provided scope: ..."` |
| Budget exists with zero allocation | `BUDGET_EXCEEDED` (409) |

## Common allocation patterns

### Flat tenant budgets

The simplest approach: allocate a single budget at the tenant level.

```
tenant:acme → allocated: 1,000,000
```

Every reservation by tenant `acme` draws from this single pool. No per-workspace or per-app limits.

### Tenant + workspace budgets

Add workspace-level budgets for environment isolation:

```
tenant:acme                          → allocated: 1,000,000
tenant:acme/workspace:production     → allocated: 500,000
tenant:acme/workspace:staging        → allocated: 200,000
tenant:acme/workspace:development    → allocated: 300,000
```

Production cannot consume more than 500,000, even if the tenant has remaining budget elsewhere.

### Per-workflow run budgets

For short-lived workflows, allocate budgets per run using the workflow field:

```
tenant:acme/workspace:production/workflow:run-12345 → allocated: 50,000
```

This caps a single workflow execution at 50,000 units.

### Per-agent budgets

For multi-agent systems, allocate per agent:

```
tenant:acme/workspace:production/agent:planner   → allocated: 100,000
tenant:acme/workspace:production/agent:executor   → allocated: 200,000
tenant:acme/workspace:production/agent:reviewer   → allocated: 50,000
```

### Custom dimensions are not budget scopes

The Subject's optional `dimensions` map (e.g., `cost_center`, `region`) does **not** participate in scope derivation. Scopes are derived only from the six standard Subject fields, in canonical order: `tenant` → `workspace` → `app` → `workflow` → `agent` → `toolset`. There is no `dimensions:` scope segment, so you cannot allocate a budget to a dimension value. v0 servers may ignore `dimensions` for budgeting decisions entirely (they only have to accept and round-trip it) — use dimensions for reporting and policy taxonomies, and model any dimension you need to *enforce* as one of the six standard fields instead.

## Updating budget configuration

Use `PATCH /v1/admin/budgets?scope={scope}&unit={unit}` to update mutable budget properties without re-creating the ledger:

```bash
curl -s -X PATCH "http://localhost:7979/v1/admin/budgets?scope=tenant:acme&unit=USD_MICROCENTS" \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -d '{
    "overdraft_limit": { "amount": 500000, "unit": "USD_MICROCENTS" },
    "commit_overage_policy": "ALLOW_WITH_OVERDRAFT",
    "metadata": { "cost_center": "engineering" }
  }' | jq .
```

You can update:

- **`overdraft_limit`** — maximum allowed debt. When changed, `is_over_limit` is atomically recalculated.
- **`commit_overage_policy`** — per-ledger overage policy override (`REJECT`, `ALLOW_IF_AVAILABLE`, `ALLOW_WITH_OVERDRAFT`).
- **`metadata`** — key-value pairs for external references (replaces the full metadata object).

Fields not included in the request are left unchanged. Returns `404` if the budget does not exist, `403` for tenant mismatch, and `409` if the budget is `CLOSED`.

## Freezing and unfreezing budgets

*New in v0.1.25.6.*

Use freeze to immediately halt all new reservations against a budget without deleting or modifying it. This is useful during incident investigations, compliance holds, or when a runaway agent is detected.

::: tip Freeze from the dashboard
Freeze and unfreeze are also one-click actions on the Budgets page in the [Cycles Admin Dashboard](/quickstart/deploying-the-cycles-dashboard) — typically faster during an active incident than crafting a curl. The dashboard also exposes an **Emergency Freeze (tenant-wide)** action that sequentially freezes every ACTIVE budget for a tenant with a confirm + blast-radius summary.
:::

### Freeze

```bash
curl -s -X POST "http://localhost:7979/v1/admin/budgets/freeze?scope=tenant:acme&unit=USD_MICROCENTS" \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -d '{"reason": "Investigating runaway agent in support workflow"}' | jq .
```

All new live reservations (and events) against the frozen scope fail with `409` and `error: BUDGET_FROZEN`. Only `/v1/decide` and dry-run reserve surface the condition as a `200` response with `decision: DENY` and `reason_code: BUDGET_FROZEN`. Fund operations also return `409` while frozen. The freeze gate applies to *new* reservations — the spec does not block committing or releasing reservations that were already active when the freeze landed. Emits a `budget.frozen` webhook event.

### Unfreeze

```bash
curl -s -X POST "http://localhost:7979/v1/admin/budgets/unfreeze?scope=tenant:acme&unit=USD_MICROCENTS" \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -d '{"reason": "Investigation complete — root cause was prompt loop, now fixed"}' | jq .
```

Transitions FROZEN → ACTIVE. Reservations resume immediately. Emits a `budget.unfrozen` webhook event. Returns 409 if the budget is already active or closed.

::: tip When to freeze vs. adjust budget
**Freeze** when you need to stop all activity immediately while investigating. The budget allocation and history are preserved. **Adjust the budget** (PATCH or fund) when you want to change how much is available. Freeze is an operational control; budget adjustment is a financial control.
:::

## Adjusting budget allocation

### Increasing a budget

Increase the `allocated` value to give a scope more room. This takes effect immediately — the next reservation check will use the new value.

### Decreasing a budget

Decrease the `allocated` value. If the new value is less than `spent + reserved`, existing reservations are not affected, but new reservations may be denied.

### Resizing a budget (RESET)

To **change the allocated ceiling** while preserving consumption history (`spent`, `reserved`, `debt`), use `RESET`:

```bash
curl -X POST "http://localhost:7979/v1/admin/budgets/fund?scope=tenant:acme&unit=USD_MICROCENTS" \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "operation": "RESET",
    "amount": { "amount": 1500000, "unit": "USD_MICROCENTS" },
    "idempotency_key": "resize-acme-q2",
    "reason": "Plan upgrade — Pro tier"
  }'
```

`RESET` sets `allocated = amount` and recalculates `remaining = amount - reserved - spent - debt`. Spent stays where it was. Use this for **plan changes, policy tightening, ceiling adjustments** — the typical "this customer moved to a bigger plan" or "we're tightening this team's limit" scenarios.

Release active reservations first if you're shrinking the ceiling below `spent + reserved` and want a clean cutover.

::: warning RESET is for resizing, not period boundaries
For a fresh billing period (clearing consumption), use `RESET_SPENT` below. A same-amount `RESET` on an exhausted budget is a no-op — `spent` stays at its old value, so `remaining` stays at 0.
:::

### Starting a new billing period (RESET_SPENT)

To **start a new billing period** — clearing accumulated spend so the scope can transact fresh — use `RESET_SPENT`:

```bash
curl -X POST "http://localhost:7979/v1/admin/budgets/fund?scope=tenant:acme&unit=USD_MICROCENTS" \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "operation": "RESET_SPENT",
    "amount": { "amount": 1000000, "unit": "USD_MICROCENTS" },
    "idempotency_key": "reset-march-2026",
    "reason": "Monthly billing period reset — March 2026"
  }'
```

`RESET_SPENT` sets `allocated = amount`, **clears spent to 0**, and preserves `reserved` (active reservations straddle the period boundary and will land in the new period's spent when they commit) and `debt` (period boundaries don't forgive debt — use `REPAY_DEBT` to clear it explicitly).

#### Optional `spent` override

For migrations, prorated signups, and corrections, supply an explicit `spent`:

```bash
# Migration: import an existing customer with their consumption already reflected
curl -X POST "http://localhost:7979/v1/admin/budgets/fund?scope=tenant:acme&unit=USD_MICROCENTS" \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "operation": "RESET_SPENT",
    "amount": { "amount": 1000000, "unit": "USD_MICROCENTS" },
    "spent":  { "amount": 400000,  "unit": "USD_MICROCENTS" },
    "idempotency_key": "migrate-acme-from-billing-vendor",
    "reason": "Imported from billing vendor — current period 40% consumed"
  }'
```

The `spent` field is honoured **only** for `RESET_SPENT`. Common patterns:

| Scenario | `spent` value | Notes |
|---|---|---|
| Routine billing-period rollover | omit (defaults to 0) | The 90% case. |
| Migration from another billing system | actual current consumption | Customer arrives with history; reflect it. |
| Prorated mid-period signup | `allocated × (days_remaining / period_days)` | New customer joins partway through. |
| Credit-back / compensation | reduced consumption value | Refund a portion after a service incident. |
| State correction | corrected value | Fix a miscounted `spent` from an upstream bug. |

Constraints:
- `spent` must be `>= 0`.
- The unit must match the budget's unit.
- The audit log records whether `spent` was explicitly supplied or defaulted to 0, distinguishing routine rollovers from operator-initiated consumption adjustments for compliance review.

##### What `remaining` looks like after RESET_SPENT

In the common case — no outstanding `debt`, no active `reserved`, `spent` omitted — `RESET_SPENT(amount=X)` produces `allocated = X` and `remaining = X`. A clean fresh period.

`remaining` can start the new period **negative** in two specific situations:

- **Carryover.** Preserved `debt` (and/or active `reserved`) exceed the new `allocated`. Periods don't forgive debt by design — use `REPAY_DEBT` if you want to clear it. Example: old period ended with `debt=1200`; `RESET_SPENT(amount=1000)` yields `remaining = 1000 - 0 - 0 - 1200 = -200`.
- **Explicit override.** You pass `spent` larger than `allocated - reserved - debt`. Example: migrating a customer already partway through a period with `RESET_SPENT(amount=1000, spent=1200)` yields `remaining = -200`.

Both cases are valid ledger states, not errors. The response returns the negative value, and the invariant `remaining = allocated - spent - reserved - debt` holds.

##### Recovery pattern: truly starting fresh when the prior period ended in debt

`RESET_SPENT` preserves `debt` by design — periods don't silently forgive obligations. If you want a customer to start the new period with a clean slate (no carryover debt, full ceiling available), pair `REPAY_DEBT` with `RESET_SPENT`:

```bash
# Prior period ended with: allocated=1000, spent=1000, debt=200, remaining=0

# Step 1: Clear the outstanding debt (e.g., after the customer paid their invoice).
curl -X POST "https://admin.example.com/v1/admin/budgets/fund?tenant_id=acme&scope=tenant:acme&unit=USD_MICROCENTS" \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -d '{
    "operation": "REPAY_DEBT",
    "amount": { "amount": 200, "unit": "USD_MICROCENTS" },
    "idempotency_key": "repay-acme-before-rollover",
    "reason": "invoice paid"
  }'
# State now: allocated=1000, spent=1000, debt=0, remaining=0

# Step 2: Start the new billing period with a fresh ceiling.
curl -X POST "https://admin.example.com/v1/admin/budgets/fund?tenant_id=acme&scope=tenant:acme&unit=USD_MICROCENTS" \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -d '{
    "operation": "RESET_SPENT",
    "amount": { "amount": 1000, "unit": "USD_MICROCENTS" },
    "idempotency_key": "rollover-acme-after-repay",
    "reason": "monthly rollover"
  }'
# State now: allocated=1000, spent=0, debt=0, remaining=1000
```

Order matters: if you skip step 1, the carryover `debt` will make the new period's `remaining` start negative. That's the correct behaviour for "customer still owes from last period", but it's not what you want if the debt has already been settled externally. Run `REPAY_DEBT` first whenever you want the next period to begin at the full ceiling.

#### Event emission

`RESET_SPENT` emits `budget.reset_spent` (distinct from `budget.reset`) so dashboards and webhook handlers can route period boundaries separately from resize events. The payload's `spent_override_provided` boolean flags which mode was used.

::: info Why budgets cannot be deleted
The admin API has no delete endpoint for budgets. A budget ledger is the permanent audit record for all spend within a scope — committed reservations reference it, and historical balances are derived from it. Deleting a ledger would create orphaned transactions and break spend reporting.

To decommission a budget: `RESET` its allocation to zero (or `DEBIT` the remaining balance). No new reservations will be approved against a zero-balance scope. The ledger stays in the system for historical queries but has no operational cost.
:::

### Funding after overdraft

If a scope has accumulated debt through `ALLOW_WITH_OVERDRAFT` commits, repay it:

```bash
curl -X POST "http://localhost:7979/v1/admin/budgets/fund?scope=tenant:acme&unit=USD_MICROCENTS" \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "operation": "REPAY_DEBT",
    "amount": { "amount": 500000, "unit": "USD_MICROCENTS" },
    "idempotency_key": "repay-001"
  }'
```

While `debt > 0` and no `overdraft_limit` is configured, new reservations against that scope are blocked with `DEBT_OUTSTANDING`. When an `overdraft_limit > 0` is set, debt within the limit does not block new reservations.

## Monitoring budgets

Use the `GET /v1/balances` endpoint to check budget state:

```bash
curl -s "http://localhost:7878/v1/balances?tenant=acme" \
  -H "X-Cycles-API-Key: your-api-key"
```

This returns the current state for all scopes matching the filter:

```json
{
  "balances": [
    {
      "scope": "tenant:acme",
      "scope_path": "tenant:acme",
      "remaining": { "amount": 750000, "unit": "USD_MICROCENTS" },
      "allocated": { "amount": 1000000, "unit": "USD_MICROCENTS" },
      "spent": { "amount": 200000, "unit": "USD_MICROCENTS" },
      "reserved": { "amount": 50000, "unit": "USD_MICROCENTS" },
      "debt": { "amount": 0, "unit": "USD_MICROCENTS" },
      "overdraft_limit": { "amount": 0, "unit": "USD_MICROCENTS" },
      "is_over_limit": false
    }
  ]
}
```

Key fields to monitor:

- **remaining** — how much room is left
- **reserved** — how much is currently held by active reservations
- **debt** — any overdraft accumulation
- **is_over_limit** — whether the scope is blocked

## Managing policies

Policies define stored caps, rate limits, and behavioral overrides matched by scope pattern.

::: warning v0 limitation
In v0, the protocol server (port 7878) does not evaluate admin-defined policies when processing reservations, commits, or events. Enforcement is planned for a future version. Today, the only policy-like behavior enforced at runtime is the `overage_policy` resolved from the request or the tenant's `default_commit_overage_policy`; budget ledgers and tenant defaults supply the rest of runtime governance. Use admin policies to model intended governance state and prepare for future enforcement.
:::

Create a policy record for matching scopes:

```bash
curl -s -X POST http://localhost:7979/v1/admin/policies \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "name": "production-limits",
    "scope_pattern": "tenant:acme-corp/workspace:production/*",
    "priority": 10,
    "commit_overage_policy": "REJECT",
    "rate_limits": {
      "max_reservations_per_minute": 100,
      "max_commits_per_minute": 100
    },
    "caps": {
      "max_tokens": 4096
    }
  }' | jq .
```

### Updating a policy

Use `PATCH /v1/admin/policies/{policy_id}` to modify mutable fields without re-creating the policy:

```bash
curl -s -X PATCH "http://localhost:7979/v1/admin/policies/$POLICY_ID" \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "priority": 20,
    "caps": { "max_tokens": 8192 },
    "rate_limits": {
      "max_reservations_per_minute": 200,
      "max_commits_per_minute": 200
    }
  }' | jq .
```

You can update: `name`, `description`, `priority`, `caps`, `commit_overage_policy`, `reservation_ttl_override`, `rate_limits`, `effective_from`, `effective_until`, and `status`. Fields not included in the request are left unchanged. Set `status` to `DISABLED` to deactivate a policy without deleting it.

## Summary

Budget allocation in Cycles:

- Is set per scope independently
- Is enforced atomically across the full scope hierarchy for each reservation
- Can be adjusted at any time with immediate effect
- Requires explicit allocation at every scope level you want to control
- Supports flat, hierarchical, per-run, and per-agent patterns (scopes derive from the six standard Subject fields; `dimensions` never derive scopes — they serve reporting, taxonomies, and policy/quota uses)

## Next steps

- [Tenants, Scopes, and Budgets](/how-to/understanding-tenants-scopes-and-budgets-in-cycles) — how tenants, scopes, and budgets work together as a unified model
- [Tenant Creation and Management](/how-to/tenant-creation-and-management-in-cycles) — create and configure tenants before allocating budgets
- [Querying Balances](/protocol/querying-balances-in-cycles-understanding-budget-state) — detailed balance query guide
- [Debt and Overdraft](/protocol/debt-overdraft-and-the-over-limit-model-in-cycles) — how overdraft affects allocation
- [How Scope Derivation Works](/protocol/how-scope-derivation-works-in-cycles) — how scopes are derived from Subjects



# Budget Templates

Ready-to-use setup scripts for common Cycles deployment patterns. Each template creates the tenant, budgets, and API key in one shot. Pick the pattern that matches your deployment, adjust the dollar amounts, and run.

**Prerequisites:**
- Cycles server running ([deployment guide](/quickstart/deploying-the-full-cycles-stack))
- `ADMIN_KEY` set to your admin API key
- `CYCLES_API_KEY` set to an API key with `budgets:write` permission

## Template 1: Single-Tenant (One Organization)

**Use when:** You have one organization, one set of agents, and need basic cost control. The simplest starting point.

**What it creates:**
- 1 tenant
- 1 API key (for your application)
- 1 USD budget at the tenant level ($100/month)
- 1 workspace budget for production ($80/month — leaves $20 for staging/dev)

```bash
#!/bin/bash
# Template 1: Single-Tenant Budget Setup
# Adjust: TENANT, MONTHLY_BUDGET_USD, PROD_BUDGET_USD

ADMIN_URL="http://localhost:7979"
TENANT="my-company"
MONTHLY_BUDGET_USD=100    # $100/month total
PROD_BUDGET_USD=80        # $80/month for production

# Convert to microcents (1 USD = 100,000,000 microcents)
TENANT_BUDGET=$((MONTHLY_BUDGET_USD * 100000000))
PROD_BUDGET=$((PROD_BUDGET_USD * 100000000))

echo "=== Creating tenant ==="
curl -s -X POST "$ADMIN_URL/v1/admin/tenants" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"tenant_id\": \"$TENANT\", \"name\": \"My Company\"}"
echo

echo "=== Creating API key ==="
curl -s -X POST "$ADMIN_URL/v1/admin/api-keys" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"tenant_id\": \"$TENANT\",
    \"name\": \"app-server\",
    \"permissions\": [\"reservations:create\", \"reservations:commit\", \"reservations:release\", \"reservations:extend\", \"reservations:list\", \"balances:read\"]
  }"
echo -e "\n>>> Save the key_secret value above — it won't be shown again\n"

echo "=== Creating tenant-level budget (\$${MONTHLY_BUDGET_USD}/month) ==="
curl -s -X POST "$ADMIN_URL/v1/admin/budgets" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"scope\": \"tenant:$TENANT\",
    \"unit\": \"USD_MICROCENTS\",
    \"allocated\": {\"amount\": $TENANT_BUDGET, \"unit\": \"USD_MICROCENTS\"}
  }"
echo

echo "=== Creating production workspace budget (\$${PROD_BUDGET_USD}/month) ==="
curl -s -X POST "$ADMIN_URL/v1/admin/budgets" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"scope\": \"tenant:$TENANT/workspace:production\",
    \"unit\": \"USD_MICROCENTS\",
    \"allocated\": {\"amount\": $PROD_BUDGET, \"unit\": \"USD_MICROCENTS\"}
  }"
echo

echo "=== Done ==="
echo "Tenant: $TENANT"
echo "Tenant budget: \$$MONTHLY_BUDGET_USD/month"
echo "Production budget: \$$PROD_BUDGET_USD/month"
echo "Remaining for staging/dev: \$$((MONTHLY_BUDGET_USD - PROD_BUDGET_USD))/month"
```

**Monthly reset (cron):**

```bash
# Add to crontab: 0 0 1 * * /path/to/reset-budget.sh
ADMIN_URL="http://localhost:7979"

curl -s -X POST "$ADMIN_URL/v1/admin/budgets/fund?scope=tenant:my-company&unit=USD_MICROCENTS" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"operation\": \"RESET_SPENT\",
    \"amount\": {\"amount\": 10000000000, \"unit\": \"USD_MICROCENTS\"},
    \"idempotency_key\": \"reset-$(date +%Y-%m)\",
    \"reason\": \"Monthly budget reset\"
  }"
```

---

## Template 2: Multi-Tenant SaaS (Customer Per Tenant)

**Use when:** You're building a SaaS product where each customer gets their own AI agent access with isolated budgets. The most common production pattern.

**What it creates per customer:**
- 1 tenant per customer
- 1 API key per customer
- 1 USD budget sized by plan tier
- Overdraft policy for Pro/Enterprise plans

```bash
#!/bin/bash
# Template 2: Multi-Tenant SaaS — Customer Onboarding
# Run once per new customer. Adjust: CUSTOMER_ID, PLAN

ADMIN_URL="http://localhost:7979"
CUSTOMER_ID="${1:?Usage: $0 <customer-id> <plan>}"
PLAN="${2:?Usage: $0 <customer-id> <plan>}"  # free | pro | enterprise

# Plan tier budgets (microcents)
case "$PLAN" in
  free)
    BUDGET=500000000        # $5
    OVERDRAFT=0
    OVERAGE_POLICY="REJECT"
    ;;
  pro)
    BUDGET=5000000000       # $50
    OVERDRAFT=500000000     # $5 overdraft
    OVERAGE_POLICY="ALLOW_WITH_OVERDRAFT"
    ;;
  enterprise)
    BUDGET=50000000000      # $500
    OVERDRAFT=5000000000    # $50 overdraft
    OVERAGE_POLICY="ALLOW_WITH_OVERDRAFT"
    ;;
  *)
    echo "Unknown plan: $PLAN (use: free, pro, enterprise)"
    exit 1
    ;;
esac

echo "=== Onboarding customer: $CUSTOMER_ID (plan: $PLAN) ==="

echo "--- Creating tenant ---"
curl -s -X POST "$ADMIN_URL/v1/admin/tenants" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"tenant_id\": \"$CUSTOMER_ID\", \"name\": \"$CUSTOMER_ID\"}"
echo

echo "--- Creating API key ---"
curl -s -X POST "$ADMIN_URL/v1/admin/api-keys" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"tenant_id\": \"$CUSTOMER_ID\",
    \"name\": \"$CUSTOMER_ID-app\",
    \"permissions\": [\"reservations:create\", \"reservations:commit\", \"reservations:release\", \"reservations:extend\", \"reservations:list\", \"balances:read\"]
  }"
echo -e "\n>>> Save the key_secret value above\n"

echo "--- Creating budget ledger ---"
curl -s -X POST "$ADMIN_URL/v1/admin/budgets" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"scope\": \"tenant:$CUSTOMER_ID\",
    \"unit\": \"USD_MICROCENTS\",
    \"allocated\": {\"amount\": $BUDGET, \"unit\": \"USD_MICROCENTS\"}
  }"
echo

if [ "$OVERDRAFT" -gt 0 ]; then
  echo "--- Setting overdraft policy ---"
  curl -s -X PATCH "$ADMIN_URL/v1/admin/budgets?scope=tenant:$CUSTOMER_ID&unit=USD_MICROCENTS" \
    -H "X-Admin-API-Key: $ADMIN_KEY" \
    -H "Content-Type: application/json" \
    -d "{
      \"overdraft_limit\": {\"amount\": $OVERDRAFT, \"unit\": \"USD_MICROCENTS\"},
      \"commit_overage_policy\": \"$OVERAGE_POLICY\"
    }"
  echo
fi

echo "=== Customer $CUSTOMER_ID onboarded (plan: $PLAN) ==="
```

**Usage:**
```bash
./onboard-customer.sh acme-corp pro
./onboard-customer.sh small-startup free
./onboard-customer.sh big-enterprise enterprise
```

**Plan tier reference:**

| Plan | Monthly budget | Overdraft | Overage policy |
|---|---|---|---|
| Free | $5 | $0 | REJECT |
| Pro | $50 | $5 | ALLOW_WITH_OVERDRAFT |
| Enterprise | $500 | $50 | ALLOW_WITH_OVERDRAFT |

---

## Template 3: Multi-Agent with RISK_POINTS

**Use when:** Your agents have tools with side effects (email, deploy, database mutations) and you need action-level control beyond cost. Adds a RISK_POINTS budget alongside USD.

**What it creates:**
- 1 tenant
- 1 API key
- 1 USD budget (cost control)
- 1 RISK_POINTS budget (action control — per-run)
- Per-run `RISK_POINTS` cap bounds caller-assigned exposure when every protected tool attempt uses a mandatory reservation boundary

```bash
#!/bin/bash
# Template 3: Multi-Agent with RISK_POINTS
# Adjust: TENANT, USD_BUDGET, RISK_BUDGET_PER_RUN

ADMIN_URL="http://localhost:7979"
TENANT="my-company"
USD_BUDGET=10000000000     # $100/month in microcents (1 USD = 100,000,000)
RISK_BUDGET_PER_RUN=250    # 250 RISK_POINTS per agent run

echo "=== Creating tenant ==="
curl -s -X POST "$ADMIN_URL/v1/admin/tenants" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"tenant_id\": \"$TENANT\", \"name\": \"My Company\"}"
echo

echo "=== Creating API key ==="
curl -s -X POST "$ADMIN_URL/v1/admin/api-keys" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"tenant_id\": \"$TENANT\",
    \"name\": \"app-server\",
    \"permissions\": [\"reservations:create\", \"reservations:commit\", \"reservations:release\", \"reservations:extend\", \"reservations:list\", \"balances:read\"]
  }"
echo -e "\n>>> Save the key_secret value above\n"

echo "=== Creating USD cost budget ($((USD_BUDGET / 100000000))/month) ==="
curl -s -X POST "$ADMIN_URL/v1/admin/budgets" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"scope\": \"tenant:$TENANT\",
    \"unit\": \"USD_MICROCENTS\",
    \"allocated\": {\"amount\": $USD_BUDGET, \"unit\": \"USD_MICROCENTS\"}
  }"
echo

echo "=== Creating per-run RISK_POINTS budget ($RISK_BUDGET_PER_RUN points) ==="
echo "    (Create this scope per agent run with the run ID in the scope path)"
echo "    Example: tenant:$TENANT/workflow:run-{uuid}"
echo
echo "    Use this curl as a template for your application code:"
cat << 'EXAMPLE'

# In your application, before each agent run:
RUN_ID="run-$(uuidgen)"
curl -s -X POST "$ADMIN_URL/v1/admin/budgets" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"scope\": \"tenant:my-company/workflow:$RUN_ID\",
    \"unit\": \"RISK_POINTS\",
    \"allocated\": {\"amount\": 250, \"unit\": \"RISK_POINTS\"}
  }"

EXAMPLE

echo "=== Tool scoring reference ==="
echo "    Assign these RISK_POINTS when reserving for each tool call:"
echo
echo "    search / read:           0 points (Tier 0)"
echo "    save_draft / log:        1 point  (Tier 1)"
echo "    external API / webhook:  5 points (Tier 2)"
echo "    send_email / update_db: 40 points (Tier 3, with 2x multiplier)"
echo "    deploy / payment:      150 points (Tier 4, with 3x multiplier)"
echo
echo "    Full scoring guide: https://runcycles.io/how-to/assigning-risk-points-to-agent-tools"
echo

echo "=== Done ==="
echo "USD budget: \$$(( USD_BUDGET / 100000000 ))/month (tenant-level)"
echo "RISK_POINTS budget: $RISK_BUDGET_PER_RUN points per run (create per run)"
```

**How the two budgets work together:**

| Budget | Scope | Resets | Controls |
|---|---|---|---|
| USD (cost) | `tenant:my-company` | Monthly via cron | Total API spend across all agents |
| RISK_POINTS (action) | `tenant:my-company/workflow:run-{uuid}` | Per run (new scope each time) | What tools the agent can use within one execution |

The USD budget bounds submitted spend. The `RISK_POINTS` budget separately bounds caller-assigned exposure. In this example, a mandatory handler could reserve 40 points per authorized email attempt, so six attempts consume 240 of 250 points and a seventh does not fit. Tool and argument authorization remains a host responsibility.

---

## Choosing a template

| Template | Best for | Adds | Complexity |
|---|---|---|---|
| **1. Single-Tenant** | Internal tools, solo teams, prototypes | Cost control | Low |
| **2. Multi-Tenant SaaS** | Customer-facing products with plan tiers | Cost control + tenant isolation | Medium |
| **3. Multi-Agent + RISK_POINTS** | Agents with side effects (email, deploy) | Cost + action control | Medium |

**Start with Template 1** to validate your integration. Upgrade to Template 2 when you add customers, or Template 3 when your agents need action-level governance.

## Next steps

- [Common Budget Patterns](/how-to/common-budget-patterns) — deeper patterns beyond these templates
- [Assigning RISK_POINTS to Tools](/how-to/assigning-risk-points-to-agent-tools) — scoring your tools for Template 3
- [Multi-Tenant SaaS Guide](/how-to/multi-tenant-saas-with-cycles) — full multi-tenant architecture
- [Budget Allocation and Management](/how-to/budget-allocation-and-management-in-cycles) — funding, resetting, overdraft policies
- [Shadow Mode Rollout](/how-to/shadow-mode-in-cycles-how-to-roll-out-budget-enforcement-without-breaking-production) — validate budgets before enforcing



# Choosing the Right Integration Pattern

Each Cycles SDK offers multiple integration patterns. This guide helps you pick the right one for your use case.

## Decision tree

<DecisionTree />

## Pattern comparison

| Pattern | Languages | Best for | Streaming | Auto-heartbeat | Auto-commit |
|---|---|---|---|---|---|
| **MCP Server** | Any (agent-native) | Cooperative budget-tool exposure in MCP hosts | — | — | — |
| **Agent framework plugin** | Python, TypeScript | Agent SDKs with lifecycle hooks | — | Yes | Yes |
| **Decorator / HOF** | Python `@cycles`, TS `withCycles`, Java `@Cycles` | Simple function calls | No | Yes | Yes |
| **Streaming adapter** | Python `stream_reservation`, TS `reserveForStream` | Streaming responses | Yes | Yes | Manual |
| **Middleware** | Express, FastAPI | Per-request budget in web apps | Both | Depends | Manual |
| **Programmatic client** | All languages | Full control, complex flows | Both | Manual | Manual |

## Pattern 0: MCP Server (zero-code tool exposure)

If your agent runs in an MCP-compatible host — Claude Desktop, Claude Code, Cursor, or Windsurf — you can expose Cycles tools without an SDK integration. This is cooperative: the standalone MCP server does not automatically wrap or block the host's other tools.

```bash
# Claude Code
claude mcp add \
  --transport stdio \
  --env CYCLES_API_KEY=cyc_live_... \
  --env CYCLES_BASE_URL=http://localhost:7878 \
  cycles \
  -- npx -y @runcycles/mcp-server
```

The agent may call `cycles_reserve`, `cycles_commit`, and other tools as part of its reasoning. No application code wraps the LLM call, so this alone is not a hard limit. Use **Cycles Budget Guard for Claude Code** or a mandatory handler, gateway, harness, or service boundary when the protected action must not bypass the reservation.

**Use when:**
- The agent host supports MCP
- You want budget awareness with zero code changes
- Cooperative, model-managed budget lifecycle is acceptable

**Don't use when:**
- You're building a non-agent application (web API, batch pipeline)
- You need a hard limit but cannot add a mandatory host or application boundary

See [Getting Started with the MCP Server](/quickstart/getting-started-with-the-mcp-server) for setup instructions.

## Pattern 0a: Agent framework plugin

For agent frameworks that expose lifecycle hooks, a plugin implements the framework's hook interface to create reservations on start and commit on end — covering the entire agent run automatically with no per-function decoration.

| Framework | Plugin / package | Hook surface |
|---|---|---|
| OpenAI Agents SDK | `runcycles_openai_agents.CyclesRunHooks` | `RunHooks` interface |
| OpenClaw | Plugin hooks | `before_model_resolve`, `before_tool_call`, etc. |
| **LangChain 1.x** (`langchain.agents.create_agent`) | [**`langchain-runcycles`**](https://pypi.org/project/langchain-runcycles/) — `CyclesModelGate`, `CyclesToolGate`, `CyclesFanOutGate` | `wrap_model_call`, `wrap_tool_call`, `before_model` (`AgentMiddleware` API) |

```python
# LangChain 1.x agent middleware
from langchain.agents import create_agent
from langchain_runcycles import CyclesFanOutGate, CyclesToolGate
from runcycles import Action, Subject

agent = create_agent(
    model="claude-sonnet-4-6",
    tools=[...],
    middleware=[
        CyclesFanOutGate(max_turns=20, client=client, subject=Subject(tenant="acme"), action=Action(kind="model.turn", name="research")),
        CyclesToolGate(client, subject=Subject(tenant="acme"), action={"send_email": Action(kind="tool.call", name="send_email")}, mode="decide"),
    ],
)
```

```python
# OpenAI Agents SDK
from agents import Agent
from runcycles_openai_agents import CyclesRunHooks

hooks = CyclesRunHooks(
    tenant="acme",
    tool_estimates={"send_email": 50, "search": 0},  # default unit: RISK_POINTS
)
result = await hooks.run(agent, input="...")
```

**Use when:**
- You're using an agent framework with lifecycle hooks (OpenAI Agents SDK, OpenClaw, LangChain 1.x `create_agent`)
- You want automatic budget governance on LLM and tool calls, plus best-effort handoff audit events
- You need tool-level risk mapping (different costs per tool)
- You want agent handoff tracking in the Cycles ledger

**Don't use when:**
- You're not using an agent framework (use `@cycles` decorator instead)
- You need per-function control over estimation and commit (use programmatic client)
- You're using bare LangChain (`ChatOpenAI`, chains, RAG) without `create_agent` — use the [LangChain callback handler](/how-to/integrating-cycles-with-langchain#callback-handler-for-non-agent-runnables) instead

See [Integrating with LangChain](/how-to/integrating-cycles-with-langchain) (Python agent middleware), [OpenAI Agents](/how-to/integrating-cycles-with-openai-agents), or [OpenClaw](/how-to/integrating-cycles-with-openclaw).

## Pattern 1: Decorator / Higher-Order Function

The simplest pattern. Wrap a function and let the SDK handle the full reserve-execute-commit lifecycle.

::: code-group
```python [Python]
@cycles(estimate=2000000, action_kind="llm.completion", action_name="gpt-4o")
def ask(prompt: str) -> str:
    return openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
    ).choices[0].message.content
```
```typescript [TypeScript]
const ask = withCycles(
  { estimate: 2000000, actionKind: "llm.completion", actionName: "gpt-4o" },
  async (prompt: string) => {
    const res = await openai.chat.completions.create({
      model: "gpt-4o",
      messages: [{ role: "user", content: prompt }],
    });
    return res.choices[0].message.content;
  },
);
```
```java [Java]
@Cycles(estimate = "2000000", actionKind = "llm.completion", actionName = "gpt-4o")
public String ask(String prompt) {
    return callOpenAI(prompt);
}
```
:::

**Use when:**
- The function makes one LLM/API call and returns a result
- You don't need to stream the response
- You want minimal code changes

**Don't use when:**
- The function streams output (use a streaming adapter instead)
- You need to control when the commit happens (use programmatic client)

Note that the decorator *can* commit with actual token-derived costs: both SDKs accept an `actual` callable that receives the function's return value — `@cycles(estimate=..., actual=lambda result: len(result) * 5)` in Python, `withCycles({ estimate: ..., actual: (result) => result.usage.total_tokens * 10 }, ...)` in TypeScript.

## Pattern 2: Streaming adapter

For streaming responses where the function returns before the stream finishes.

### TypeScript (`reserveForStream`)

```typescript
const handle = await reserveForStream({
  client: cyclesClient,
  estimate: 5000000,
  actionKind: "llm.completion",
  actionName: "gpt-4o",
});

try {
  const stream = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: prompt }],
    stream: true,
  });

  let inputTokens = 0, outputTokens = 0;
  for await (const chunk of stream) {
    // ... consume stream, track tokens ...
  }

  await handle.commit(actualCost, { tokensInput: inputTokens, tokensOutput: outputTokens });
} catch (err) {
  await handle.release("stream_error");
  throw err;
}
```

### Python (`stream_reservation`)

The Python client's equivalent is `client.stream_reservation(...)` — a context manager that reserves on enter and commits (or releases, on exception) on exit:

```python
with client.stream_reservation(
    action=Action(kind="llm.completion", name="gpt-4o"),
    estimate=Amount(unit=Unit.USD_MICROCENTS, amount=5_000_000),
    cost_fn=lambda u: u.tokens_input * 250 + u.tokens_output * 1000,
) as reservation:
    for chunk in stream:
        # ... consume stream, track tokens ...
        reservation.usage.add_output_tokens(chunk_tokens)
# Auto-committed on success, auto-released on exception.
```

**Use when:**
- The LLM response is streamed to the client
- You need to track token counts from stream events
- You want automatic heartbeat during streaming

**Don't use when:**
- The response is not streamed (use `withCycles` instead — simpler)

## Pattern 3: Middleware

For web applications where every request needs budget governance.

::: code-group
```typescript [Express]
// cyclesGuard is an example pattern (not exported by the SDK) — build it from
// the programmatic client; see Integrating Cycles with Express for the full source.
app.post("/api/chat", cyclesGuard({ client, actionKind: "llm.completion", ... }), handler);
```
```python [FastAPI]
@app.post("/api/chat")
@cycles(estimate=2000000, action_kind="llm.completion", action_name="gpt-4o")
async def chat(request: ChatRequest):
    ...
```
:::

See [Integrating Cycles with Express](/how-to/integrating-cycles-with-express) for a complete `cyclesGuard` middleware implementation.

**Use when:**
- Budget enforcement should apply to every request on a route
- You want your application to translate Cycles' HTTP 409 `BUDGET_EXCEEDED`
  response into an HTTP 402 response for callers
- Budget should be scoped per-request (e.g., per-tenant)

**Don't use when:**
- Budget logic varies significantly between requests on the same route
- You're not in a web framework context

## Pattern 4: Programmatic client

Full control over the reserve-commit lifecycle. Use this when no higher-level pattern fits.

::: code-group
```python [Python]
client = CyclesClient(config)

reservation = client.create_reservation({
    "idempotency_key": "req-001",
    "subject": {"tenant": "acme-corp"},
    "action": {"kind": "llm.completion", "name": "gpt-4o"},
    "estimate": {"amount": 2000000, "unit": "USD_MICROCENTS"},
    "ttl_ms": 30000,
})

if reservation.status == 409:
    # Live (non-dry-run) denials are HTTP 409 BUDGET_EXCEEDED — there is
    # no decision field to check (decision=DENY only appears when dry_run=true).
    handle_denial()
elif reservation.is_success:
    reservation_id = reservation.body["reservation_id"]
    result = call_llm()
    client.commit_reservation(reservation_id, {
        "idempotency_key": "commit-001",
        "actual": {"amount": actual_cost, "unit": "USD_MICROCENTS"},
    })
```
```typescript [TypeScript]
// The TypeScript client sends and receives wire-format (snake_case) JSON.
const reservation = await client.createReservation({
  idempotency_key: "req-001",
  subject: { tenant: "acme-corp" },
  action: { kind: "llm.completion", name: "gpt-4o" },
  estimate: { amount: 2000000, unit: "USD_MICROCENTS" },
  ttl_ms: 30000,
});

if (reservation.status === 409) {
  // Live (non-dry-run) denials are HTTP 409 BUDGET_EXCEEDED — there is
  // no decision field to check (decision=DENY only appears when dry_run=true).
  handleDenial();
} else if (reservation.isSuccess) {
  const reservationId = reservation.getBodyAttribute("reservation_id") as string;
  const result = await callLLM();
  await client.commitReservation(reservationId, {
    idempotency_key: "commit-001",
    actual: { amount: actualCost, unit: "USD_MICROCENTS" },
  });
}
```
:::

**Use when:**
- You need to inspect the reservation decision before proceeding
- You need to commit with exact actual token counts
- You're building a custom integration layer
- You need to manage TTL extensions manually
- The operation spans multiple steps with different commit points

**Don't use when:**
- A decorator or streaming adapter would work — they handle heartbeat, retry, and cleanup automatically

## Combining patterns

In practice, most applications use multiple patterns:

```python
# Simple calls — decorator
@cycles(estimate=500000, action_kind="llm.completion", action_name="gpt-4o-mini")
def classify(text: str) -> str:
    ...

# Complex flows — programmatic
async def agent_loop(task: str):
    client = CyclesClient(config)
    while not done:
        reservation = client.create_reservation(...)
        result = call_tool(...)
        client.commit_reservation(...)
```

## Next steps

- [Getting Started with the MCP Server](/quickstart/getting-started-with-the-mcp-server) — zero-code runtime authority for Claude / Cursor / Windsurf
- [Integrating with OpenAI Agents](/how-to/integrating-cycles-with-openai-agents) — budget governance for OpenAI Agents SDK
- [Getting Started with Python](/quickstart/getting-started-with-the-python-client)
- [Getting Started with TypeScript](/quickstart/getting-started-with-the-typescript-client)
- [Getting Started with Spring Boot](/quickstart/getting-started-with-the-cycles-spring-boot-starter)
- [Handling Streaming Responses](/how-to/handling-streaming-responses-with-cycles)



# Choosing the Right Overage Policy

Cycles provides three overage policies that control what happens when actual usage exceeds the reserved estimate. This guide helps you pick the right one based on your use case.

For detailed mechanics and implementation, see [Commit Overage Policies](/protocol/commit-overage-policies-in-cycles-reject-allow-if-available-and-allow-with-overdraft).

## Decision flowchart

Ask these questions in order:

1. **Has the work already happened?** (external import, retroactive accounting, side effect already fired)
   → **ALLOW_WITH_OVERDRAFT** — the ledger must reflect reality. Note that debt is bounded by the scope's `overdraft_limit`: if the commit would push debt beyond it, the server rejects it with `409 OVERDRAFT_LIMIT_EXCEEDED`, so size the limit for your import volumes

2. **Can you estimate costs reliably?** (fixed-price APIs, known token counts, deterministic operations)
   → **REJECT** with a 10–20% buffer — hard enforcement is safe when estimates are tight

3. **Everything else** — variable-cost LLM calls, tool invocations, streaming responses, multi-step agents
   → **ALLOW_IF_AVAILABLE** (the default) — does not reject a valid commit merely because of overage, never creates debt, and caps the charged delta at available budget

## By use case

### LLM completions (GPT-4, Claude, Gemini)

**Recommended:** ALLOW_IF_AVAILABLE

Token counts vary by prompt, context window, and model behavior. Estimation is inherently imprecise. For an instrumented call whose commit passes ownership, state, expiry, and unit validation, ALLOW_IF_AVAILABLE records the submitted actual but caps the charged overage when budget runs low.

```python
@cycles(estimate=50000, action_kind="llm.completion", action_name="openai:gpt-4o")
def call_llm(prompt: str) -> str:
    return openai_client.chat(prompt)
```

### Tool invocations with known costs

**Recommended:** REJECT or ALLOW_IF_AVAILABLE

If a tool call has a fixed, predictable cost (e.g., a search API at $0.01/query), REJECT with a buffer works well. If cost varies, use ALLOW_IF_AVAILABLE.

```python
# Fixed cost — REJECT is safe
@cycles(estimate=1000, overage_policy="REJECT", action_kind="tool.search", action_name="google-search")
def search(query: str) -> list:
    return search_api.query(query)
```

### External usage imports

**Recommended:** ALLOW_WITH_OVERDRAFT

When importing usage from an external billing system, the work already happened. The budget ledger must reflect it regardless of remaining budget.

```python
from runcycles import EventCreateRequest, Subject, Action, Amount

# Recording usage from an external gateway
client.create_event(EventCreateRequest(
    idempotency_key="import-ext-001",
    subject=Subject(tenant="acme", app="gateway"),
    action=Action(kind="llm.completion", name="external:model"),
    actual=Amount(amount=usage_amount, unit="USD_MICROCENTS"),
    overage_policy="ALLOW_WITH_OVERDRAFT"
))
```

### Multi-step agent workflows

**Recommended:** ALLOW_IF_AVAILABLE

Agents that chain multiple LLM calls and tool invocations have highly variable total cost. ALLOW_IF_AVAILABLE lets each step commit without risk of rejection, while the budget boundary naturally stops new reservations when funds run out.

### Multi-tenant platforms

**Recommended:** ALLOW_IF_AVAILABLE as the tenant default

Set `default_commit_overage_policy: ALLOW_IF_AVAILABLE` at the tenant level. This gives tenants a safe default — no debt, no rejected commits — while individual requests can override to REJECT or ALLOW_WITH_OVERDRAFT as needed.

### SLA-critical operations

**Recommended:** ALLOW_WITH_OVERDRAFT

When an operation must succeed even when the budget is exhausted — critical alerts, compliance actions, safety-related tool calls — use ALLOW_WITH_OVERDRAFT so budget exhaustion alone does not block the action. This is not unlimited: debt is bounded by the scope's `overdraft_limit`, and a commit that would push debt beyond that limit is rejected with `409 OVERDRAFT_LIMIT_EXCEEDED`. Set an `overdraft_limit` large enough to cover your worst-case SLA-critical burst, and monitor debt so it is repaid before the limit is reached.

### Background batch processing

**Recommended:** REJECT

For batch jobs where individual items can be retried, REJECT provides hard budget control. If an item exceeds budget, skip it and retry later when budget is replenished.

## Mixing policies

Most production systems use different policies for different action classes:

| Action class | Policy | Why |
|---|---|---|
| LLM completions | ALLOW_IF_AVAILABLE | Variable cost, always record |
| Fixed-cost tools | REJECT | Predictable, retry-safe |
| External imports | ALLOW_WITH_OVERDRAFT | Already happened |
| Agent orchestration | ALLOW_IF_AVAILABLE | Variable, multi-step |
| Safety-critical ops | ALLOW_WITH_OVERDRAFT | Not blocked by exhaustion (up to `overdraft_limit`) |

## Setting the default

The overage policy resolves in this order (most specific wins):

1. **Request-level** `overage_policy` field — per-call override on the reservation or event
2. **Policy-level** — pattern-based override configured via the Admin API
3. **Budget ledger-level** `commit_overage_policy` — override for a specific scope's ledger
4. **Tenant default** `default_commit_overage_policy` — set via Admin API

If none of these is set, the server falls back to `ALLOW_IF_AVAILABLE`.

For most teams, leaving the default as ALLOW_IF_AVAILABLE and overriding per-request for specific action classes is the simplest approach.

## Summary

- **REJECT** — hard stops, best when estimates are reliable, may leave unaccounted gaps
- **ALLOW_IF_AVAILABLE** — safe default, always commits, caps at budget, no debt
- **ALLOW_WITH_OVERDRAFT** — ledger accuracy above all, creates debt, requires reconciliation

When in doubt, use ALLOW_IF_AVAILABLE. It handles the widest range of scenarios without operational overhead.

## Next steps

- [Commit Overage Policies](/protocol/commit-overage-policies-in-cycles-reject-allow-if-available-and-allow-with-overdraft) — detailed mechanics and concurrency semantics
- [Cost Estimation Cheat Sheet](/how-to/cost-estimation-cheat-sheet) — how estimation strategy ties to overage policy
- [Debt, Overdraft, and the Over-Limit Model](/protocol/debt-overdraft-and-the-over-limit-model-in-cycles) — understanding debt and reconciliation
- [How Events Work](/protocol/how-events-work-in-cycles-direct-debit-without-reservation) — direct debit without reservation



# Client Performance Tuning

The default client configuration works well for most workloads. This guide covers when and how to tune for high-throughput or latency-sensitive scenarios.

::: tip When to tune
Consider tuning when you observe any of these:
- Concurrent reservation count regularly exceeds 100
- Client-side p99 latency exceeds 50ms
- Commit retry rate exceeds 5%
- Connection timeout errors in client logs

For server-side benchmarks and baseline expectations, see the [Performance Benchmarks](/blog/cycles-server-performance-benchmarks) blog post.
:::

## Timeout tuning

All four clients configure two timeouts: connection (TCP handshake) and read (waiting for server response).

### Default values

| Setting | Python | TypeScript | Spring Boot | Rust |
|---|---|---|---|---|
| Connect timeout | `connect_timeout=2.0` (seconds) | `connectTimeout=2000` (ms) | `cycles.http.connect-timeout=2s` | `connect_timeout=2s` |
| Read timeout | `read_timeout=5.0` (seconds) | `readTimeout=5000` (ms) | `cycles.http.read-timeout=5s` | `read_timeout=5s` |

::: info TypeScript timeout behavior
Node's built-in `fetch` does not distinguish connection from read timeout. `connectTimeout` and `readTimeout` are summed into a single `AbortSignal.timeout()` value (default: 7000ms total).
:::

::: warning Heartbeat timeout budget
Automatic heartbeat reserves two complete request-attempt budgets plus a safety margin inside the remaining lease. Keep the enforced attempt timeout well below half the shortest expected TTL. The current defaults produce a finite 7-second attempt budget in TypeScript, Spring Boot, and Rust, and a 12-second whole-attempt deadline in Python; a 30-second attempt timeout leaves no positive cadence inside the default 60-second lease.
:::

### Tuning profiles

Choose a profile based on your deployment topology:

**Co-located (client and server on same network):**

```python
# Python
config = CyclesConfig(
    base_url="http://cycles-server:7878",
    api_key="cyc_live_...",
    connect_timeout=0.5,   # 500ms — same-network handshake is fast
    read_timeout=2.0,      # 2s — server p99 is <5ms, leave headroom
)
```

```typescript
// TypeScript
const config = new CyclesConfig({
  baseUrl: "http://cycles-server:7878",
  apiKey: "cyc_live_...",
  connectTimeout: 500,
  readTimeout: 2000,
});
```

```yaml
# Spring Boot
cycles:
  http:
    connect-timeout: 500ms
    read-timeout: 2s
```

```rust
// Rust
use runcycles::CyclesClient;
use std::time::Duration;

let client = CyclesClient::builder("cyc_live_...", "http://cycles-server:7878")
    .connect_timeout(Duration::from_millis(500))
    .read_timeout(Duration::from_secs(2))
    .build();
```

**Cross-region or high-latency network:**

```python
# Python
config = CyclesConfig(
    base_url="https://cycles.us-east.example.com",
    api_key="cyc_live_...",
    connect_timeout=5.0,   # TLS handshake across regions
    read_timeout=15.0,     # account for network jitter
)
```

```yaml
# Spring Boot
cycles:
  http:
    connect-timeout: 5s
    read-timeout: 15s
```

**High-throughput with aggressive retry:**

```python
# Python — fail fast, retry quickly
config = CyclesConfig(
    base_url="http://cycles-server:7878",
    api_key="cyc_live_...",
    connect_timeout=1.0,
    read_timeout=3.0,
    retry_max_attempts=10,
    retry_initial_delay=0.1,
    retry_multiplier=1.5,
    retry_max_delay=5.0,
)
```

::: warning
Never set the read timeout below the server's expected p99 latency. At default load, the server's p99 for a reserve-commit cycle is ~20ms. Under heavy load it can reach 50ms+. A read timeout of 100ms will cause spurious failures.
:::

## Connection pooling

### Python (httpx)

The Python client uses `httpx.Client`, which manages a connection pool automatically. Key details:

- Connections are reused across requests (HTTP keep-alive)
- Pool timeout is hardcoded at 5 seconds (time to acquire a connection from the pool)
- Write timeout is hardcoded at 5 seconds
- httpx uses its default limits: 100 max connections, of which up to 20 are kept alive for reuse — with no per-host cap (all connections can go to the single Cycles server)

The constructor accepts only a `CyclesConfig` — there is no parameter for injecting a custom httpx transport or client, so the pool limits are not tunable per client. For most workloads, the defaults are sufficient. If you see `PoolTimeout` errors, you have more than 100 concurrent requests in flight on a single client instance. Solutions:

1. **Use multiple client instances** — partition by tenant or workload so each instance carries a share of the concurrency (each gets its own 100-connection pool).

2. **Bound your concurrency** — cap in-flight Cycles calls below the pool size (e.g. an `asyncio.Semaphore` around reservation calls, or a bounded worker pool).

### TypeScript (fetch)

The TypeScript client uses Node's built-in `fetch`, which relies on the runtime's HTTP agent for connection reuse:

- Node.js 20+ reuses connections automatically via its global `undici` agent
- No explicit pool configuration is exposed
- Keep-alive is enabled by default

For high-throughput Node.js services, ensure you're running Node 20+ where `fetch` connection reuse is reliable.

### Spring Boot (Reactor Netty)

The Spring Boot starter uses `WebClient` backed by Reactor Netty's `HttpClient`. Reactor Netty manages its own connection pool:

- Default pool: Reactor Netty's shared global pool — max connections is `2 × max(available processors, 8)` (so 16 on small hosts), with no idle timeout by default
- The 45-second default is the **pending-acquire timeout**: how long a request waits to acquire a connection from the pool before failing, not an idle timeout
- Configure via Reactor Netty system properties (`reactor.netty.pool.maxConnections`, `reactor.netty.pool.acquireTimeout`) or provide a custom `WebClient` bean

To customize the connection pool:

```java
@Bean
public WebClient cyclesWebClient(CyclesProperties props) {
    ConnectionProvider provider = ConnectionProvider.builder("cycles")
        .maxConnections(200)
        .pendingAcquireTimeout(Duration.ofSeconds(10))  // default: 45s
        .maxIdleTime(Duration.ofSeconds(30))
        .build();

    HttpClient httpClient = HttpClient.create(provider)
        .option(ChannelOption.CONNECT_TIMEOUT_MILLIS,
            (int) props.getHttp().getConnectTimeout().toMillis())
        .responseTimeout(props.getHttp().getReadTimeout());

    return WebClient.builder()
        .clientConnector(new ReactorClientHttpConnector(httpClient))
        .baseUrl(props.getBaseUrl())
        .defaultHeader("X-Cycles-API-Key", props.getApiKey())
        .build();
}
```

## Retry strategy tuning

The settlement retry engine handles transport failures, 5xx responses, and rate limits with same-key recovery. Known actual usage is also journaled before the first settlement request, so retry tuning changes how quickly the client retries; it does not remove restart durability while journaling remains enabled. Two common profiles:

### Fast-fail (latency-sensitive paths)

Limit in-process retry work when prolonged recovery is undesirable. In Rust this directly bounds the inline guarded-commit wait; the Python, TypeScript, and Spring lifecycle helpers schedule settlement recovery in the background.

| Setting | Python | TypeScript | Spring Boot | Rust |
|---|---|---|---|---|
| Max attempts | `retry_max_attempts=2` | `retryMaxAttempts: 2` | `cycles.retry.max-attempts=2` | `.retry_max_attempts(2)` |
| Initial delay | `retry_initial_delay=0.1` | `retryInitialDelay: 100` | `cycles.retry.initial-delay=100ms` | `.retry_initial_delay(Duration::from_millis(100))` |
| Multiplier | `retry_multiplier=1.5` | `retryMultiplier: 1.5` | `cycles.retry.multiplier=1.5` | `.retry_multiplier(1.5)` |
| Max delay | `retry_max_delay=1.0` | `retryMaxDelay: 1000` | `cycles.retry.max-delay=1s` | `.retry_max_delay(Duration::from_secs(1))` |
| Scheduled backoff | ~250ms | ~250ms | ~250ms | ~250ms |

### Aggressive convergence (must-commit workloads)

Retry aggressively because the action already happened and the ledger must reflect it.

| Setting | Python | TypeScript | Spring Boot | Rust |
|---|---|---|---|---|
| Max attempts | `retry_max_attempts=10` | `retryMaxAttempts: 10` | `cycles.retry.max-attempts=10` | `.retry_max_attempts(10)` |
| Initial delay | `retry_initial_delay=0.2` | `retryInitialDelay: 200` | `cycles.retry.initial-delay=200ms` | `.retry_initial_delay(Duration::from_millis(200))` |
| Multiplier | `retry_multiplier=1.5` | `retryMultiplier: 1.5` | `cycles.retry.multiplier=1.5` | `.retry_multiplier(1.5)` |
| Max delay | `retry_max_delay=60.0` | `retryMaxDelay: 60000` | `cycles.retry.max-delay=60s` | `.retry_max_delay(Duration::from_secs(60))` |
| Scheduled backoff | ~23 seconds | ~23 seconds | ~23 seconds | ~23 seconds |

Scheduled backoff excludes the time spent in the initial request and each retry attempt. A valid server `Retry-After` can also lengthen the schedule according to the SDK's bounded-delay policy.

### When to disable retry

```python
config = CyclesConfig(base_url="...", api_key="...", retry_enabled=False)
```

Disable retry when:
- You handle retries at a higher level (e.g. job queue with built-in retry)
- You need deterministic latency with no background work
- Testing, where retry masks failures

Disabling retry does not disable the durable journal. Pending known-actual settlement remains available for replay on a later client start. Disable journaling only when the higher-level system durably persists the exact settlement body and idempotency key and implements equivalent replay and expiry fallback.

## Anti-patterns

### Creating a new client per request

Every `CyclesClient` instance creates a new HTTP connection pool. Creating one per request wastes connections and prevents reuse.

```python
# BAD — new client (and connection pool) for every call
@cycles(estimate=1000, client=CyclesClient(config))
def process(text: str) -> str:
    return call_llm(text)
```

```python
# GOOD — reuse a single client via module default
client = CyclesClient(config)
set_default_client(client)

@cycles(estimate=1000)
def process(text: str) -> str:
    return call_llm(text)
```

The same applies in TypeScript (`setDefaultClient`) and Spring Boot (the auto-configured `CyclesClient` bean is a singleton).

### Expensive estimate computations in the hot path

The estimate callable runs synchronously before each reservation. Keep it fast:

```python
# BAD — network call in the estimate
@cycles(estimate=lambda text: fetch_token_count_from_api(text))
def process(text: str) -> str:
    return call_llm(text)
```

```python
# GOOD — pre-compute or use a fast heuristic
@cycles(estimate=lambda text: len(text) // 4)  # ~4 characters per token
def process(text: str) -> str:
    return call_llm(text)
```

## High-throughput checklist

1. **Reuse a single client instance** across all requests (all 3 clients)
2. **Warm up on startup** — make a lightweight call through the client so its connection pool is established before real traffic arrives:
   ```python
   client = CyclesClient(config)
   client.get_balances(tenant="my-tenant")  # any cheap read warms the pool
   ```
   To check server availability without an API key, use the public readiness probe (`GET /actuator/health/readiness`). Note that the aggregate `/actuator/health` and other actuator endpoints require the admin key since server 0.1.25.45 — only the liveness and readiness probes are public.
3. **Graceful shutdown** — commit or release active reservations before process exit
4. **Pre-compute estimates** outside the decorator/HOF hot path
5. **Lower timeouts** if co-located, raise if cross-region
6. **Use durable retry** for must-commit workloads
7. **Monitor** retry rates and timeout errors — rising rates signal infrastructure issues

## Server-side tuning

For server-side performance:

- **Redis connection pool** — default 128 connections. See [Production Operations Guide](/how-to/production-operations-guide).
- **Expiry sweep interval** — default 5000ms. See [Server Configuration Reference](/configuration/server-configuration-reference-for-cycles).
- **Benchmarks** — Reserve 5.3ms p50, 2,632 reserve-commit lifecycles/sec at 32 threads. See [Performance Benchmarks](/blog/cycles-server-performance-benchmarks).

## Next steps

- [Production Operations Guide](/how-to/production-operations-guide) — server infrastructure and Redis tuning
- [Monitoring and Alerting](/how-to/monitoring-and-alerting) — metrics and alerting setup
- [Observability Setup](/how-to/observability-setup) — Prometheus, Grafana, and Datadog integration
- [Python Client Configuration](/configuration/python-client-configuration-reference) — all Python config options
- [TypeScript Client Configuration](/configuration/typescript-client-configuration-reference) — all TypeScript config options
- [Spring Client Configuration](/configuration/client-configuration-reference-for-cycles-spring-boot-starter) — all Spring Boot config options



# Common Budget Patterns

Practical recipes for common budget governance scenarios. Each pattern shows the scope hierarchy and budget allocation needed.

::: tip Need cost estimates?
See the [Cost Estimation Cheat Sheet](/how-to/cost-estimation-cheat-sheet) for per-model pricing and how to translate token counts into `USD_MICROCENTS`.
:::

## Per-user daily budgets

Give each user a daily spending limit.

**Scope:** `tenant:acme-corp/workspace:prod/app:chatbot/agent:{user_id}`

```bash
# Create a $5/day budget for user-123
curl -s -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "scope": "tenant:acme-corp/workspace:prod/app:chatbot/agent:user-123",
    "unit": "USD_MICROCENTS",
    "allocated": { "amount": 500000000, "unit": "USD_MICROCENTS" }
  }' | jq .
```

**Reset daily** with a cron job or scheduled task — use `RESET_SPENT` so
each day's spent clears to 0 (not `RESET`, which preserves spent and
would leave the budget exhausted):

```bash
# Reset each user's daily budget to $5, clearing yesterday's spend
curl -s -X POST ".../fund" \
  -d '{"operation": "RESET_SPENT", "amount": {"amount": 500000000, "unit": "USD_MICROCENTS"}, ...}'
```

**In your app:**

```python
@cycles(
    estimate=2000000,
    action_kind="llm.completion",
    action_name="gpt-4o",
    # Callable: re-evaluated on every call, so each request resolves
    # the current user (a bare `current_user.id` would be captured
    # once at decoration time)
    agent=lambda prompt: current_user.id,
)
def chat(prompt: str) -> str:
    ...
```

## Per-conversation session budgets

Cap spending per conversation to prevent runaway loops.

**Scope:** `tenant:acme-corp/workflow:{conversation_id}`

```bash
# Create a $0.50 budget per conversation
curl -s -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "scope": "tenant:acme-corp/workflow:conv-abc-123",
    "unit": "USD_MICROCENTS",
    "allocated": { "amount": 50000000, "unit": "USD_MICROCENTS" }
  }' | jq .
```

**In your app:**

```python
@cycles(
    estimate=2000000,
    action_kind="llm.completion",
    action_name="gpt-4o",
    # Callable receives the function's arguments at call time
    workflow=lambda conversation_id, message: conversation_id,
)
def reply(conversation_id: str, message: str) -> str:
    ...
```

When the conversation budget runs out, the next call is denied. The user sees a "budget exhausted" message and can start a new conversation (with its own fresh budget).

## Model-tier budgets

Different budget pools for different model tiers. Prevents expensive model calls from consuming the cheap-model budget.

**Scopes:**

```
tenant:acme-corp/app:chatbot/toolset:tier-premium    → $50/month
tenant:acme-corp/app:chatbot/toolset:tier-standard    → $200/month
tenant:acme-corp/app:chatbot/toolset:tier-economy     → $500/month
```

**In your app:**

```python
MODEL_TIERS = {
    "gpt-4o": "tier-premium",
    "claude-sonnet": "tier-premium",
    "gpt-4o-mini": "tier-standard",
    "claude-haiku": "tier-economy",
}

@cycles(
    estimate=2000000,
    action_kind="llm.completion",
    # Callables: resolved per call from the function's arguments
    action_name=lambda model_name, prompt: model_name,
    toolset=lambda model_name, prompt: MODEL_TIERS[model_name],
)
def call_model(model_name: str, prompt: str) -> str:
    ...
```

## Team-level rollup budgets

Give each team its own budget while also enforcing a company-wide cap.

**Scopes (both need budgets):**

```
tenant:acme-corp                          → $10,000/month (company cap)
tenant:acme-corp/workspace:engineering    → $5,000/month
tenant:acme-corp/workspace:marketing      → $2,000/month
tenant:acme-corp/workspace:support        → $3,000/month
```

A reservation with `tenant=acme-corp, workspace=engineering` checks budget at both levels. If the engineering team has budget but the company is at its cap, the reservation is denied.

## Agent loop with per-run budget

Cap the total cost of a single agent run to prevent runaway loops.

```bash
# Create a $2 budget for this specific run
curl -s -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "scope": "tenant:acme-corp/workflow:run-xyz-789",
    "unit": "USD_MICROCENTS",
    "allocated": { "amount": 200000000, "unit": "USD_MICROCENTS" }
  }' | jq .
```

**In your app:**

```python
def agent_run(task: str, run_id: str):
    while not done:
        @cycles(
            estimate=2000000,
            action_kind="llm.completion",
            action_name="gpt-4o",
            workflow=run_id,
        )
        def think(prompt: str) -> str:
            return call_llm(prompt)

        try:
            result = think(next_prompt)
            # ... process result, decide next step ...
        except BudgetExceededError:
            return "Agent stopped: budget limit for this run reached."
```

## Gradual degradation pattern

Use multiple budget thresholds to degrade gracefully instead of hard-stopping.

**Budget scopes with different allocations:**

```
tenant:acme-corp/app:chatbot                    → $100 (hard limit)
tenant:acme-corp/app:chatbot/toolset:premium    → $60  (premium model threshold)
tenant:acme-corp/app:chatbot/toolset:tools      → $40  (tool use threshold)
```

**In your app:**

```python
from runcycles import BudgetExceededError, cycles

@cycles(estimate=5000000, action_kind="llm.completion",
        action_name="gpt-4o", toolset="premium")
def premium_response(prompt: str) -> str:
    return call_gpt4o(prompt)

@cycles(estimate=200000, action_kind="llm.completion",
        action_name="gpt-4o-mini")
def economy_response(prompt: str) -> str:
    return call_gpt4o_mini(prompt)

def respond(prompt: str) -> str:
    # Try premium model first
    try:
        return premium_response(prompt)
    except BudgetExceededError:
        pass  # Premium budget exhausted, fall through

    # Fall back to cheap model
    try:
        return economy_response(prompt)
    except BudgetExceededError:
        return "All budgets exhausted. Please try again later."
```

## Multi-tenant SaaS with per-customer budgets

Each customer gets an isolated budget. Use tenant-per-customer or workspace-per-customer depending on your isolation model.

**Option A: Tenant per customer** (strongest isolation — separate API keys)

```
tenant:customer-a → $500/month
tenant:customer-b → $200/month
```

**Option B: Workspace per customer** (shared tenant, simpler management)

```
tenant:my-saas/workspace:customer-a → $500/month
tenant:my-saas/workspace:customer-b → $200/month
```

The app resolves the scope from the authenticated request:

```python
@cycles(
    estimate=2000000,
    action_kind="llm.completion",
    action_name="gpt-4o",
    # Callable: resolves the customer per call
    workspace=lambda request: request.customer_id,
)
def handle_request(request):
    ...
```

## Next steps

- [Tenants, Scopes, and Budgets](/how-to/understanding-tenants-scopes-and-budgets-in-cycles) — how the three building blocks fit together
- [Tenant, Workflow, and Run Budgets](/how-to/how-to-model-tenant-workflow-and-run-budgets-in-cycles) — detailed multi-level budgeting guide
- [Budget Allocation and Management](/how-to/budget-allocation-and-management-in-cycles) — funding operations
- [Scope Derivation](/protocol/how-scope-derivation-works-in-cycles) — how hierarchical scopes work
- [AI Agent Budget Patterns: A Practical Guide](/blog/agent-budget-patterns-visual-guide) — architectural thinking behind each budget pattern



# Cost Estimation Cheat Sheet

This guide answers the most common question when adopting Cycles: **how much should I reserve for a given LLM call?**

For the broader strategy guide on estimation approaches, see [Estimate Exposure Before Execution](/how-to/how-to-estimate-exposure-before-execution-practical-reservation-strategies-for-cycles).

## The unit: USD_MICROCENTS

Cycles uses **USD_MICROCENTS** as its primary currency unit:

```
1 USD_MICROCENT = 10⁻⁶ cents = 10⁻⁸ dollars
$1.00 = 100,000,000 microcents
$0.01 = 1,000,000 microcents
```

The formula for converting provider pricing to microcents:

```
microcents = (price_per_million_tokens / 1,000,000) × token_count × 100,000,000
```

Simplified:

```
microcents = price_per_million_tokens × token_count × 100
```

## Provider pricing reference

### OpenAI

| Model | Input (per 1M tokens) | Output (per 1M tokens) | Input (microcents/token) | Output (microcents/token) |
|---|---|---|---|---|
| gpt-5.6-sol | $5.00 | $30.00 | 500 | 3,000 |
| gpt-5.6-terra | $2.50 | $15.00 | 250 | 1,500 |
| gpt-5.6-luna | $1.00 | $6.00 | 100 | 600 |
| gpt-5 | $1.25 | $10.00 | 125 | 1,000 |
| gpt-5-mini | $0.25 | $2.00 | 25 | 200 |
| gpt-5-nano | $0.05 | $0.40 | 5 | 40 |
| gpt-4o | $2.50 | $10.00 | 250 | 1,000 |
| gpt-4o-mini | $0.15 | $0.60 | 15 | 60 |
| gpt-4.1 | $2.00 | $8.00 | 200 | 800 |
| gpt-4.1-mini | $0.40 | $1.60 | 40 | 160 |
| gpt-4.1-nano | $0.10 | $0.40 | 10 | 40 |
| o3 | $2.00 | $8.00 | 200 | 800 |
| o3-mini | $1.10 | $4.40 | 110 | 440 |
| o4-mini | $1.10 | $4.40 | 110 | 440 |

### Anthropic

| Model | Input (per 1M tokens) | Output (per 1M tokens) | Input (microcents/token) | Output (microcents/token) |
|---|---|---|---|---|
| Claude Opus 4.8 | $5.00 | $25.00 | 500 | 2,500 |
| Claude Sonnet 4.6 | $3.00 | $15.00 | 300 | 1,500 |
| Claude Haiku 4.5 | $1.00 | $5.00 | 100 | 500 |

### Google

| Model | Input (per 1M tokens) | Output (per 1M tokens) | Input (microcents/token) | Output (microcents/token) |
|---|---|---|---|---|
| Gemini 2.5 Pro | $1.25 | $10.00 | 125 | 1,000 |
| Gemini 2.5 Flash | $0.30 | $2.50 | 30 | 250 |
| Gemini 2.5 Flash-Lite | $0.10 | $0.40 | 10 | 40 |

### Groq on-demand

| Model | Input (per 1M tokens) | Output (per 1M tokens) | Input (microcents/token) | Output (microcents/token) |
|---|---|---|---|---|
| openai/gpt-oss-20b | $0.075 | $0.30 | 7.5 | 30 |
| openai/gpt-oss-120b | $0.15 | $0.60 | 15 | 60 |
| qwen/qwen3.6-27b | $0.60 | $3.00 | 60 | 300 |

> Open-model pricing and availability vary by host. The table above is specifically Groq's on-demand pricing, not a universal rate for those model families. Round the final reservation amount up when a per-token conversion is fractional. Self-hosted models have no provider token invoice, but still consume compute; use a unit that matches what you want to bound. See the [Groq integration guide](/how-to/integrating-cycles-with-groq) and [Ollama integration guide](/how-to/integrating-cycles-with-ollama).

::: info Note
Provider rates above were checked on July 24, 2026. The OpenAI table includes the current GPT-5.6 family plus selected older models that still appear in examples. GPT-5.6 cache reads are discounted, cache writes cost 1.25 times the uncached input rate, and requests above 272,000 input tokens use higher rates for the full request. Prices, caching rules, long-context tiers, and regional premiums change; check each provider's pricing page before deploying. The formulas remain the same.
:::

## Quick estimation formula

For a single LLM call:

```
estimate = (max_input_tokens × input_microcents) + (max_output_tokens × output_microcents)
```

Then add a safety buffer:

```
reservation_amount = estimate × 1.2   # 20% buffer
```

### Example: GPT-5.6 Luna call with 2,000 input tokens, 1,000 max output tokens

```
input_cost  = 2,000 × 100 = 200,000 microcents
output_cost = 1,000 × 600 = 600,000 microcents
total       = 800,000 microcents ($0.008)
with buffer = 960,000 microcents
```

### Example: Claude Sonnet 4.6 call with 4,000 input tokens, 2,000 max output tokens

```
input_cost  = 4,000 × 300   = 1,200,000 microcents
output_cost = 2,000 × 1,500 = 3,000,000 microcents
total       = 4,200,000 microcents ($0.042)
with buffer = 5,040,000 microcents
```

## Estimation helpers in code

::: code-group
```python [Python]
import math

# Simple cost estimator
def estimate_cost(input_tokens: int, max_output_tokens: int, model: str) -> int:
    """Return estimated cost in USD_MICROCENTS with 20% buffer."""
    rates = {
        "gpt-5.6-sol":     (500, 3000),
        "gpt-5.6-terra":   (250, 1500),
        "gpt-5.6-luna":    (100, 600),
        "gpt-4o":          (250, 1000),
        "gpt-4o-mini":     (15, 60),
        "gpt-4.1":         (200, 800),
        "gpt-4.1-mini":    (40, 160),
        "gpt-4.1-nano":    (10, 40),
        "claude-sonnet":   (300, 1500),
        "claude-haiku":    (100, 500),
        "gemini-2.5-pro":  (125, 1000),
        "gemini-2.5-flash":(30, 250),
        "groq:gpt-oss-20b":(7.5, 30),
        "groq:gpt-oss-120b":(15, 60),
        "groq:qwen3.6-27b": (60, 300),
    }
    input_rate, output_rate = rates.get(model, (100, 600))
    estimate = (input_tokens * input_rate) + (max_output_tokens * output_rate)
    return math.ceil(estimate * 1.2)

# Usage with the @cycles decorator
@cycles(
    estimate=lambda prompt, max_tokens=1000: estimate_cost(
        len(prompt) // 4, max_tokens, "gpt-5.6-luna"
    ),
    action_kind="llm.completion",
    action_name="openai:gpt-5.6-luna",
)
def ask(prompt: str, max_tokens: int = 1000) -> str:
    ...
```
```typescript [TypeScript]
function estimateCost(inputTokens: number, maxOutputTokens: number, model: string): number {
  const rates: Record<string, [number, number]> = {
    "gpt-5.6-sol":     [500, 3000],
    "gpt-5.6-terra":   [250, 1500],
    "gpt-5.6-luna":    [100, 600],
    "gpt-4o":          [250, 1000],
    "gpt-4o-mini":     [15, 60],
    "gpt-4.1":         [200, 800],
    "gpt-4.1-mini":    [40, 160],
    "gpt-4.1-nano":    [10, 40],
    "claude-sonnet":   [300, 1500],
    "claude-haiku":    [100, 500],
    "gemini-2.5-pro":  [125, 1000],
    "gemini-2.5-flash":[30, 250],
    "groq:gpt-oss-20b":[7.5, 30],
    "groq:gpt-oss-120b":[15, 60],
    "groq:qwen3.6-27b": [60, 300],
  };
  const [inputRate, outputRate] = rates[model] ?? [100, 600];
  const estimate = inputTokens * inputRate + maxOutputTokens * outputRate;
  return Math.ceil(estimate * 1.2);
}

const ask = withCycles(
  {
    estimate: (prompt: string) => estimateCost(Math.ceil(prompt.length / 4), 1000, "gpt-5.6-luna"),
    actionKind: "llm.completion",
    actionName: "openai:gpt-5.6-luna",
  },
  async (prompt: string) => { ... },
);
```
:::

## Common reservation amounts

Quick reference for typical operations (including 20% buffer):

| Operation | Model | Typical Estimate (microcents) | Approx USD |
|---|---|---|---|
| Short chat reply (500 in / 200 out) | gpt-5.6-luna | 204,000 | $0.002 |
| Long chat reply (2,000 in / 1,000 out) | gpt-5.6-luna | 960,000 | $0.010 |
| Document summary (8,000 in / 2,000 out) | gpt-5.6-luna | 2,400,000 | $0.024 |
| Short chat reply (500 in / 200 out) | gpt-4o-mini | 23,400 | $0.0002 |
| Long chat reply (2,000 in / 1,000 out) | claude-sonnet | 2,520,000 | $0.025 |
| Code generation (4,000 in / 4,000 out) | claude-sonnet | 8,640,000 | $0.086 |

## When you don't know the exact token count

Use these rules of thumb:

- **1 token is roughly 4 characters** of English text (or ~0.75 words)
- For input: count the prompt characters and divide by 4
- For output: use the `max_tokens` parameter you're passing to the provider
- **Always round up** — over-reserving temporarily locks budget but releases the unused portion on commit

## Using TOKENS unit instead of USD_MICROCENTS

If you prefer to budget in tokens rather than dollars:

```python
@cycles(estimate=2000, unit="TOKENS", action_kind="llm.completion", action_name="gpt-5.6-luna")
def ask(prompt: str) -> str:
    ...
```

This is simpler but does not account for different costs across models. Use `TOKENS` when all your calls use the same model, or when you want model-agnostic budgets.

## Overage policies and estimation

Your estimation strategy should match your [overage policy](/protocol/commit-overage-policies-in-cycles-reject-allow-if-available-and-allow-with-overdraft):

| Policy | Estimation approach |
|---|---|
| **REJECT** | Reserve conservatively (use 120-150% buffer). Under-reserving causes commit failures. |
| **ALLOW_IF_AVAILABLE** | Reserve your best estimate. If actual exceeds reserved, the delta is deducted from remaining budget. |
| **ALLOW_WITH_OVERDRAFT** | Reserve normally. Overage is allowed up to the overdraft limit. Best for SLA-critical operations. |

## Next steps

- [Estimate Exposure Before Execution](/how-to/how-to-estimate-exposure-before-execution-practical-reservation-strategies-for-cycles) — detailed strategy guide for improving estimation over time
- [Understanding Units](/protocol/understanding-units-in-cycles-usd-microcents-tokens-credits-and-risk-points) — how USD_MICROCENTS, TOKENS, CREDITS, and RISK_POINTS work
- [Commit Overage Policies](/protocol/commit-overage-policies-in-cycles-reject-allow-if-available-and-allow-with-overdraft) — what happens when actual exceeds estimated
- [How Much Do AI Agents Actually Cost?](/blog/how-much-do-ai-agents-cost) — per-token pricing across providers with real-world cost scenarios



# Custom Field Resolvers in Cycles

The `CyclesFieldResolver` interface lets you resolve Subject fields dynamically at runtime in the Spring Boot Starter. This is useful when values like tenant, workspace, or agent depend on the current request context, user session, or database lookup.

::: info Python equivalent
In the Python client, Subject fields are resolved from decorator parameters, then from `CyclesConfig` defaults. For dynamic resolution, pass a `CyclesConfig` with fields set at initialization time, or pass subject fields directly to each `@cycles` decorator call. See the [Python Client Configuration Reference](/configuration/python-client-configuration-reference) for details.
:::

## The interface

```java
@FunctionalInterface
public interface CyclesFieldResolver {
    String resolve();
}
```

A resolver returns a `String` value for its associated Subject field, or `null` if no value should be set.

## How resolution works

For each Subject field (tenant, workspace, app, workflow, agent, toolset), the starter resolves the value in this order:

1. **Annotation attribute** — if set on the `@Cycles` annotation, it wins
2. **Configuration property** — if set in `application.yml` (e.g., `cycles.tenant`)
3. **CyclesFieldResolver bean** — if a Spring bean named after the field exists

This means a resolver is the fallback. It is only called when the annotation and configuration do not provide a value.

::: tip Value already in a method argument?
Since 0.2.1, subject fields on `@Cycles` accept SpEL directly — `@Cycles(value = "1000", tenant = "#tenantId")` pulls the tenant from a method argument without a resolver bean. Resolvers are for values that come from outside the method signature (security context, database, thread-local state).
:::

## Creating a resolver

Register a Spring bean whose name matches the Subject field you want to resolve.

### Tenant resolver

```java
@Component("tenant")
public class TenantResolver implements CyclesFieldResolver {

    @Autowired
    private TenantContext tenantContext;

    @Override
    public String resolve() {
        return tenantContext.getCurrentTenant();
    }
}
```

### Workspace resolver

```java
@Component("workspace")
public class WorkspaceResolver implements CyclesFieldResolver {

    @Autowired
    private EnvironmentService environmentService;

    @Override
    public String resolve() {
        return environmentService.getCurrentEnvironment();
    }
}
```

### Agent resolver

```java
@Component("agent")
public class AgentResolver implements CyclesFieldResolver {

    @Autowired
    private AgentRegistry registry;

    @Override
    public String resolve() {
        return registry.getCurrentAgentId();
    }
}
```

## Supported field names

Register a bean with one of these names:

| Bean name | Subject field |
|---|---|
| `"tenant"` | `subject.tenant` |
| `"workspace"` | `subject.workspace` |
| `"app"` | `subject.app` |
| `"workflow"` | `subject.workflow` |
| `"agent"` | `subject.agent` |
| `"toolset"` | `subject.toolset` |

## Real-world example: multi-tenant SaaS

In a multi-tenant application, the tenant is typically extracted from the current request (JWT token, session, or request header):

```java
@Component("tenant")
public class RequestTenantResolver implements CyclesFieldResolver {

    @Override
    public String resolve() {
        // Get tenant from Spring Security context
        Authentication auth = SecurityContextHolder.getContext().getAuthentication();
        if (auth instanceof TenantAwareAuthentication tenantAuth) {
            return tenantAuth.getTenantId();
        }
        return null;
    }
}
```

Now every `@Cycles`-annotated method automatically uses the request's tenant without specifying it in the annotation:

```java
@Cycles("5000")
public String summarize(String text) {
    // tenant is resolved automatically from the request context
    return chatModel.call(text);
}
```

## Real-world example: database lookup

If the tenant or workspace comes from a database:

```java
@Component("tenant")
public class DatabaseTenantResolver implements CyclesFieldResolver {

    @Autowired
    private RepositoryAccessService repositoryService;

    @Override
    public String resolve() {
        Optional<String> tenant = repositoryService.findTenant();
        return tenant.orElse(null);
    }
}
```

## Resolver precedence in practice

Given this configuration:

```yaml
cycles:
  tenant: default-tenant
  workspace: production
```

And this resolver:

```java
@Component("tenant")
public class TenantResolver implements CyclesFieldResolver {
    public String resolve() { return "resolved-tenant"; }
}
```

The effective values depend on the annotation:

```java
// Uses the annotation value: "explicit-tenant"
@Cycles(value = "1000", tenant = "explicit-tenant")
public void method1() { ... }

// No annotation value → uses the config value: "default-tenant".
// The resolver is never called because config already provides a value.
@Cycles("1000")
public void method2() { ... }
```

The resolver is only consulted when neither the annotation nor configuration provides a value. With the configuration above, `TenantResolver` never runs. Remove `cycles.tenant` from the configuration and `method2()` resolves to `"resolved-tenant"` from the resolver bean instead.

## Returning null

If a resolver returns `null`, that field is omitted from the Subject. The server will then derive it from context (e.g., the API key's tenant).

```java
@Component("workflow")
public class WorkflowResolver implements CyclesFieldResolver {
    @Override
    public String resolve() {
        // Only set workflow if we're inside a workflow context
        WorkflowContext ctx = WorkflowContext.current();
        return ctx != null ? ctx.getWorkflowId() : null;
    }
}
```

## Thread safety

Resolvers are called on the thread that invokes the `@Cycles`-annotated method. If your resolver reads from `ThreadLocal` state (like `SecurityContextHolder` or request-scoped beans), it will work correctly as long as the annotated method runs on the request thread.

If you use `@Async` or execute on a different thread, ensure the context is propagated.

## Testing resolvers

Test resolvers directly since they implement a simple interface:

```java
@Test
void testTenantResolution() {
    TenantResolver resolver = new TenantResolver();
    // Set up the context your resolver reads from
    TenantContext.set("test-tenant");

    assertEquals("test-tenant", resolver.resolve());
}

@Test
void testNullWhenNoContext() {
    TenantResolver resolver = new TenantResolver();
    TenantContext.clear();

    assertNull(resolver.resolve());
}
```

## Summary

- Implement `CyclesFieldResolver` and register as a named Spring bean
- Bean name must match the Subject field: `tenant`, `workspace`, `app`, `workflow`, `agent`, or `toolset`
- Resolvers are the lowest-priority source (after annotation and config)
- Return `null` to omit a field
- Useful for multi-tenant SaaS, request-scoped context, and database lookups

## Working example in the demo app

The demo application includes a complete working field resolver:

- **`CyclesTenantResolver.java`** (`cycles-demo-client-java-spring/src/main/java/io/runcycles/demo/client/spring/resolvers/CyclesTenantResolver.java`) — Registered as `@Component("tenant")`, implements `CyclesFieldResolver`, and resolves the tenant dynamically via a repository service lookup. This is exactly the "database lookup" pattern described above.

The resolver is used automatically by all `@Cycles`-annotated methods in the demo when no tenant is specified in the annotation or `application.yml` configuration.

## Next steps

- [Python Client Configuration](/configuration/python-client-configuration-reference) — Python config properties and resolution order
- [Spring Client Configuration](/configuration/client-configuration-reference-for-cycles-spring-boot-starter) — Spring Boot config properties and resolution order
- [Getting Started with the Spring Boot Starter](/quickstart/getting-started-with-the-cycles-spring-boot-starter) — annotation usage
- [Testing with Cycles](/how-to/testing-with-cycles) — testing resolvers and annotations



# Integration Ecosystem

Cycles integrates with the tools, frameworks, and AI providers you already use. Whether you're building autonomous agents, adding runtime authority to an existing application, or exploring what's possible with controlled AI spending, there's an integration path for you.

## AI Model Providers

### OpenAI

Integrate Cycles with GPT-5.6 and other models available through the OpenAI API. Reserve budget per protected request and use standard subject fields for enforceable workflow or agent scopes.

- [OpenAI integration guide (Python)](/how-to/integrating-cycles-with-openai)
- [OpenAI integration guide (TypeScript)](/how-to/integrating-cycles-with-openai-typescript)
- [openai.com](https://openai.com)

### Anthropic

Use Cycles with Claude models to set spending limits on autonomous agent workflows powered by Anthropic's API. Available in both Python and TypeScript.

- [Anthropic integration guide (Python)](/how-to/integrating-cycles-with-anthropic)
- [Anthropic integration guide (TypeScript)](/how-to/integrating-cycles-with-anthropic-typescript)
- [anthropic.com](https://anthropic.com)

### Google Gemini

Add runtime authority to applications built on Google's Gemini family of models.

- [Gemini integration guide](/how-to/integrating-cycles-with-google-gemini)
- [ai.google.dev](https://ai.google.dev)

### AWS Bedrock

Cycles works with AWS Bedrock's multi-model platform, giving you budget control across any foundation model available through Bedrock.

- [AWS Bedrock integration guide](/how-to/integrating-cycles-with-aws-bedrock)
- [aws.amazon.com/bedrock](https://aws.amazon.com/bedrock)

### Ollama / Local LLMs

Budget control for local model runners — track GPU time and compute costs for self-hosted models. Works with Ollama, vLLM, text-generation-inference, and LocalAI.

- [Ollama integration guide](/how-to/integrating-cycles-with-ollama)
- [ollama.com](https://ollama.com)

### Groq

Budget governance for Groq's LPU-accelerated inference. Uses the OpenAI-compatible API with current Groq model IDs and pricing, plus an application-owned fallback pattern after a primary route's budget reservation is rejected.

- [Groq integration guide](/how-to/integrating-cycles-with-groq)
- [groq.com](https://groq.com)

## AI Frameworks & SDKs

### LangChain (Python)

[![PyPI downloads](https://img.shields.io/pypi/dm/langchain-runcycles?label=downloads&color=555&style=flat-square)](https://pypi.org/project/langchain-runcycles/)

Build budget-aware LangChain agents in Python. The `langchain-runcycles` package ships three `AgentMiddleware` classes — `CyclesModelGate`, `CyclesToolGate`, and `CyclesFanOutGate` — that gate model calls, tool calls, and runaway agent loops in `create_agent` workflows. The parent `runcycles` SDK includes a lifecycle-managed callback recipe for bare runnables (chains and RAG), with heartbeat and durable settlement rather than an in-memory-only commit path.

- [langchain-runcycles on PyPI](https://pypi.org/project/langchain-runcycles/)
- [LangChain integration guide](/how-to/integrating-cycles-with-langchain)
- [Source on GitHub](https://github.com/runcycles/langchain-runcycles)
- [Python LangChain documentation](https://docs.langchain.com/oss/python/langchain/overview)

### LangChain.js

The same LangChain integration, purpose-built for JavaScript and TypeScript environments.

- [LangChain.js integration guide](/how-to/integrating-cycles-with-langchain-js)
- [JavaScript LangChain documentation](https://docs.langchain.com/oss/javascript/langchain/overview)

### LangGraph

Budget control for LangGraph stateful agent workflows. Use LangChain's callback handler inside graph nodes, or scope budgets per node with the `@cycles` decorator. Supports conditional routing based on remaining budget.

- [LangGraph integration guide](/how-to/integrating-cycles-with-langgraph)
- [langchain-ai.github.io/langgraph](https://langchain-ai.github.io/langgraph/)

### Vercel AI SDK

Add Cycles runtime authority to applications built with the Vercel AI SDK for seamless spending control in Next.js and other Vercel-deployed projects.

- [Vercel AI SDK integration guide](/how-to/integrating-cycles-with-vercel-ai-sdk)
- [ai-sdk.dev](https://ai-sdk.dev/)

### Spring AI

Integrate Cycles with Spring AI to bring runtime authority to Java and Kotlin AI applications. Two paths:

- **Auto-wired advisor** ([`cycles-spring-ai-starter`](https://github.com/runcycles/cycles-spring-ai-starter)) — zero-code gating of every `ChatClient.call()`. Recommended for pure Spring AI apps.
- **`@Cycles` annotation** ([`cycles-client-java-spring`](https://github.com/runcycles/cycles-spring-boot-starter)) — method-level gating with SpEL-driven estimates. Use for non-Spring-AI code paths.

See the [integration guide](/how-to/integrating-cycles-with-spring-ai) for the comparison + when to use each.

- [Spring AI integration guide](/how-to/integrating-cycles-with-spring-ai)
- [Spring AI strategic quickstart](/quickstart/how-to-add-hard-budget-limits-to-spring-ai-with-cycles)
- [spring.io/projects/spring-ai](https://spring.io/projects/spring-ai)

### LlamaIndex

Add budget governance to LlamaIndex RAG pipelines. Guard retrieval and generation stages separately for fine-grained cost control.

- [LlamaIndex integration guide](/how-to/integrating-cycles-with-llamaindex)
- [llamaindex.ai](https://www.llamaindex.ai)

### CrewAI

Budget control for CrewAI multi-agent workflows. Scope budgets per agent and per crew with hierarchical budget paths.

- [CrewAI integration guide](/how-to/integrating-cycles-with-crewai)
- [crewai.com](https://www.crewai.com)

### Pydantic AI

Guard Pydantic AI agent runs and tool calls with the `@cycles` decorator. Works with structured output and tool scoping.

- [Pydantic AI integration guide](/how-to/integrating-cycles-with-pydantic-ai)
- [Pydantic AI documentation](https://pydantic.dev/docs/ai/overview/)

### AnyAgent

Budget governance for AnyAgent's unified agent interface. A single callback covers all seven supported frameworks (OpenAI Agents, LangChain, LlamaIndex, Google, Agno, smolagents, TinyAgent) with no per-framework code.

- [AnyAgent integration guide](/how-to/integrating-cycles-with-anyagent)
- [mozilla-ai.github.io/any-agent](https://mozilla-ai.github.io/any-agent/)

### AutoGen

Budget governance for Microsoft AutoGen multi-agent workflows. Wrap the model client with Cycles reservations for per-call and per-agent cost control across teams, swarms, and graph flows.

- [AutoGen integration guide](/how-to/integrating-cycles-with-autogen)
- [microsoft.github.io/autogen](https://microsoft.github.io/autogen/)

## Web Frameworks

### Next.js

Add budget governance to Next.js applications with route-level budget guards, server actions, and client-side error handling. Works with any LLM provider.

- [Next.js integration guide](/how-to/integrating-cycles-with-nextjs)
- [nextjs.org](https://nextjs.org)

### Express.js

Add Cycles middleware to your Express.js API to enforce runtime authority on any route that triggers AI spending.

- [Express.js integration guide](/how-to/integrating-cycles-with-express)
- [expressjs.com](https://expressjs.com)

### Django

Add Cycles middleware to Django applications for budget-checked views, per-tenant isolation, and preflight budget guards.

- [Django integration guide](/how-to/integrating-cycles-with-django)
- [djangoproject.com](https://www.djangoproject.com)

### Flask

Add Cycles budget guards to Flask applications with error handlers, `before_request` hooks, and per-tenant isolation.

- [Flask integration guide](/how-to/integrating-cycles-with-flask)
- [flask.palletsprojects.com](https://flask.palletsprojects.com)

### FastAPI

Use the Cycles Python client with FastAPI for high-performance, budget-aware AI APIs.

- [FastAPI integration guide](/how-to/integrating-cycles-with-fastapi)
- [fastapi.tiangolo.com](https://fastapi.tiangolo.com)

## Agent Platforms

### MCP (Model Context Protocol)

Cycles provides an MCP server that exposes runtime authority as tools for any MCP-compatible client, including Claude Desktop, Claude Code, Cursor, and Windsurf.

- [MCP integration guide](/how-to/integrating-cycles-with-mcp)
- [modelcontextprotocol.io](https://modelcontextprotocol.io)

### OpenAI Agents SDK

[![PyPI downloads](https://img.shields.io/pypi/dm/runcycles-openai-agents?label=downloads&color=555&style=flat-square)](https://pypi.org/project/runcycles-openai-agents/)

Add budget governance to OpenAI Agents SDK workflows. The plugin hooks into the SDK's `RunHooks` interface to enforce budgets on LLM calls and tool invocations, and emits best-effort zero-amount audit events for agent handoffs — with tool risk mapping and pre-run guardrails.

- [runcycles-openai-agents on PyPI](https://pypi.org/project/runcycles-openai-agents/)
- [OpenAI Agents integration guide](/how-to/integrating-cycles-with-openai-agents)
- [Source on GitHub](https://github.com/runcycles/cycles-openai-agents)

### OpenClaw

[![npm downloads](https://img.shields.io/npm/dt/@runcycles/openclaw-budget-guard?label=downloads&color=555&style=flat-square)](https://www.npmjs.com/package/@runcycles/openclaw-budget-guard)

Connect Cycles to OpenClaw for budget-controlled multi-agent orchestration.

- [@runcycles/openclaw-budget-guard on npm](https://www.npmjs.com/package/@runcycles/openclaw-budget-guard)
- [OpenClaw integration guide](/how-to/integrating-cycles-with-openclaw)

## Official SDKs

### Python Client

[![PyPI downloads](https://img.shields.io/pypi/dm/runcycles?label=downloads&color=555&style=flat-square)](https://pypi.org/project/runcycles/)

The official Cycles Python client. Install from PyPI and start enforcing budgets in minutes.

- [runcycles on PyPI](https://pypi.org/project/runcycles/)
- [Python quickstart](/quickstart/getting-started-with-the-python-client)

### TypeScript Client

[![npm downloads](https://img.shields.io/npm/dt/runcycles?label=downloads&color=555&style=flat-square)](https://www.npmjs.com/package/runcycles)

The official Cycles TypeScript client for Node.js and browser environments.

- [runcycles on npm](https://www.npmjs.com/package/runcycles)
- [TypeScript quickstart](/quickstart/getting-started-with-the-typescript-client)

### Rust Client

[![crates.io](https://img.shields.io/crates/v/runcycles?label=crates.io&color=555&style=flat-square)](https://crates.io/crates/runcycles)

The official Cycles Rust client (currently 0.3.2). Async-first, built for budget-aware agents and services in Rust.

- [runcycles on crates.io](https://crates.io/crates/runcycles)
- [Rust quickstart](/quickstart/getting-started-with-the-rust-client)

### AP2 Payment-Mandate Guard (Python)

[![PyPI downloads](https://img.shields.io/pypi/dm/runcycles-ap2?label=downloads&color=555&style=flat-square)](https://pypi.org/project/runcycles-ap2/)

Runtime authority guard for AP2 (Agent Payments Protocol) — reserve, commit, and release around agent payment mandates. It deduplicates Cycles accounting and rejects divergent reuse of an `open_mandate_hash`; identical retries still require PSP idempotency or a separate consume-once claim to prevent duplicate charges. Works with Google's AP2 spec and any AP2-compatible SDK.

- [runcycles-ap2 on PyPI](https://pypi.org/project/runcycles-ap2/)
- [Source on GitHub](https://github.com/runcycles/cycles-ap2-python)

### MCP Server

[![npm downloads](https://img.shields.io/npm/dt/@runcycles/mcp-server?label=downloads&color=555&style=flat-square)](https://www.npmjs.com/package/@runcycles/mcp-server)

The Cycles MCP server exposes runtime authority as tools for Claude Desktop, Claude Code, Cursor, and Windsurf.

- [@runcycles/mcp-server on npm](https://www.npmjs.com/package/@runcycles/mcp-server)
- [MCP quickstart](/quickstart/getting-started-with-the-mcp-server)

### Spring Boot Starter (generic `@Cycles` AOP)

[![Maven Central](https://img.shields.io/maven-central/v/io.runcycles/cycles-client-java-spring?label=Maven%20Central&color=555&style=flat-square)](https://central.sonatype.com/artifact/io.runcycles/cycles-client-java-spring)

Auto-configured Cycles integration for Spring Boot applications using the `@Cycles` annotation with SpEL-driven cost estimates. Available on Maven Central.

- [cycles-client-java-spring on Maven Central](https://central.sonatype.com/artifact/io.runcycles/cycles-client-java-spring)
- [Spring Boot quickstart](/quickstart/getting-started-with-the-cycles-spring-boot-starter)

### Spring AI Starter (advisor-based)

[![Maven Central](https://img.shields.io/maven-central/v/io.runcycles/cycles-spring-ai-starter?label=Maven%20Central&color=555&style=flat-square)](https://central.sonatype.com/artifact/io.runcycles/cycles-spring-ai-starter)

Spring AI-specific starter that auto-wires Cycles `CallAdvisor` + `StreamAdvisor` onto every `ChatClient`, gating non-streaming and streaming LLM invocations through Cycles without code changes at call sites. Also ships `CyclesToolGate` (opt-in per-tool gating), a pluggable `SubjectResolver` for per-request tenant attribution, and an `ObservationConvention` that emits Cycles attribution + `cycles.reservation_id` on chat-client traces. Companion to the generic Spring Boot starter — depend on this for Spring AI apps.

- [cycles-spring-ai-starter on Maven Central](https://central.sonatype.com/artifact/io.runcycles/cycles-spring-ai-starter)
- [Spring AI integration guide](/how-to/integrating-cycles-with-spring-ai)

## Protocol & Standards

### Cycles Protocol

The Cycles Protocol is an open specification for runtime authority in autonomous agent systems, licensed under Apache 2.0. Build your own implementation or contribute to the spec.

- [Cycles Protocol on GitHub](https://github.com/runcycles/cycles-protocol)

### OpenAPI Specification

A complete OpenAPI specification is available for the Cycles API, making it straightforward to generate clients in any language or integrate with API tooling.

- [Interactive API Reference](/api/)

## Community Tools

The Cycles ecosystem grows with every project that adopts runtime authority. If you've built a library, plugin, tool, or integration that works with Cycles, we want to hear about it.

Building something with Cycles? Add a [Built with Cycles badge](/community/badges) to your project and let the community know what you're working on.



# Cycles Budget Guard for Claude Code

The [Cycles MCP Server](/how-to/integrating-cycles-with-mcp) gives agents budget *tools* — but honoring a `DENY` inside the agent's tool loop is cooperative: nothing in MCP forces the model to reserve before acting. **Cycles Budget Guard for Claude Code** closes that gap by putting Cycles in the **tool dispatch path**: a `PreToolUse` hook reserves budget before each gated tool call, and a denial blocks the call at the harness layer. The model cannot skip that hook decision.

Repository: [runcycles/cycles-claude-plugin](https://github.com/runcycles/cycles-claude-plugin) (Apache-2.0, zero runtime dependencies). The hooks require Node.js 22 or newer.

## Install

```
/plugin marketplace add runcycles/cycles-claude-plugin
/plugin install cycles-budget-guard@runcycles
```

Then configure the environment Claude Code runs in:

```bash
export CYCLES_BASE_URL=https://your-cycles-server
export CYCLES_API_KEY=your-key
export CYCLES_DEFAULT_TENANT=acme        # one subject field is required; tenant shown
export CYCLES_DEFAULT_APP=claude-code    # optional, finer attribution
```

The protocol requires at least one standard subject field: `tenant`, `workspace`, `app`, `workflow`, `agent`, or `toolset`. It does not specifically require `tenant`; see [Understanding Tenants, Scopes, and Budgets](/how-to/understanding-tenants-scopes-and-budgets-in-cycles) when choosing the routing subject. Without a base URL or subject default, the plugin remains dormant. Once configured, an invalid value blocks calls and identifies the affected variable.

## How Claude Code budget enforcement works

1. **`PreToolUse`** reserves a flat per-call cost. The call is **blocked** on: a Cycles `DENY`; any authoritative protocol rejection (budget exhausted, frozen, or closed; debt; auth failure; invalid request); or any malformed response. Fail-open applies only to genuine outages (5xx, network errors, a four-second timeout). A fail-open call is unmetered because no reservation exists; `CYCLES_CC_FAIL_CLOSED=true` blocks that path.
2. **Tool-list caps are enforced at the gate.** `ALLOW_WITH_CAPS` tool allowlists and denylists block violating calls, with [allowlist precedence defined by the protocol](/protocol/caps-and-the-three-way-decision-model-in-cycles). Max-token, remaining-step, and cooldown caps are added to model context as guidance rather than mechanically enforced by the hook.
3. **`PostToolUse`** (success) durably changes the reservation record from `hold` to `commit`, then attempts settlement. If the reservation expired, was finalized, or is missing, the hook submits an idempotent usage event. This recovery path begins only after the successful outcome is durably recorded.
4. **`PostToolUseFailure`** attempts to release a `hold` for a call Claude Code reports as failed and retains a failed release for retry. A failed tool can still have partial side effects, so this hook outcome is not proof that no work occurred.
5. **`SessionEnd`** retries releases, commits, and usage events for the ending session. **`SessionStart`** replays commit and usage-event records across sessions with the same routing identity, but deliberately leaves unresolved `hold` records to their owning session or server-side TTL. State is scoped per OS user to a routing hash of (base URL, full subject, unit). Different routing identities cannot see one another's records; projects using the same tuple share a recovery scope.

## Budget Guard options

| Variable | Default | Meaning |
|---|---|---|
| `CYCLES_CC_UNIT` | `CREDITS` | Unit requested for each gated reservation |
| `CYCLES_CC_COST` | `1` | Flat amount requested for each gated reservation |
| `CYCLES_CC_SKIP_TOOLS` | `^(Read\|Glob\|Grep\|LS\|NotebookRead\|TodoWrite\|AskUserQuestion)$` | Configurable tools not gated by default. Set `^$` to gate every non-Cycles tool |
| `CYCLES_CC_FAIL_CLOSED` | `false` | `true` blocks calls when the Cycles server is unreachable |
| `CYCLES_CC_TTL_MS` | `1800000` (30 min) | Reservation TTL; must outlive permission prompts and long tool runs |

The Cycles budget tools themselves are never gated (exact-namespace recursion guard). The plugin configures the pinned companion Cycles MCP Server, `@runcycles/mcp-server@0.6.1`, which `npx` fetches as needed rather than the plugin vendoring it. The model can plan with `cycles_check_balance` and explicit reserves while the dispatch hook gates other configured tools.

## Verify enforcement

1. Use a test subject with an exhausted, frozen, or closed Cycles budget.
2. Ask Claude Code to run a gated action such as a `Bash` command.
3. Confirm the reservation is denied and the tool does not execute.
4. Run `/cycles-budget-guard:budget` to inspect the active budget scopes (which reflect your configured subject).

## Failure, retry, and privacy semantics

- **Identity and retries**: when Claude Code supplies a non-empty per-call `tool_use_id`, transport retries reuse the same reservation and distinct IDs remain distinct. Older inputs without `tool_use_id` use a hash of session, prompt, tool, and input; identical fallback calls can collide and undercount.
- **Integrity vs. availability**: a reserve response the plugin cannot interpret (unknown decision, missing reservation id, mistyped caps) is treated as an integrity failure and **denied** — only outages are eligible for fail-open. A malformed settlement response retains local state for retry; state is cleared only after `COMMITTED`, `RELEASED`, or `APPLIED` is confirmed.
- **Privacy**: Cycles hook requests include the configured API credential, subject identifiers, tool name, unit/amount, reservation TTL or ID and fixed settlement reasons as needed, plus opaque idempotency digests computed locally. Raw tool arguments, file contents, and prompts are not included in Cycles request bodies; Claude Code's model-provider traffic is separate. Policy: [runcycles.io/privacy](/privacy).
- **Platform**: Claude Code only. Cowork support is deliberately unclaimed until hook and environment behavior are verified there.

## Choose the Cycles MCP Server or Budget Guard

| Capability | Cycles MCP Server alone | Cycles Budget Guard for Claude Code |
|---|---|---|
| Balance, reservation, and metering tools | Available | Available through the pinned companion server |
| Pre-execution tool blocking | Cooperative only | Hook-enforced for gated tools |
| Host support | Any compatible MCP host | Claude Code only |
| Setup | Add server | Add plugin and environment configuration |

For other hosts, see the [security model](https://github.com/runcycles/cycles-mcp-server#security-model--enforcement-boundary), the [OpenClaw integration](/how-to/integrating-cycles-with-openclaw), and the guide to [choosing an integration pattern](/how-to/choosing-the-right-integration-pattern).



# Error Handling Patterns in Cycles Client Code

This guide covers practical patterns for handling Cycles errors in your application — both with the decorator/annotation and with the programmatic client.

For Python-specific patterns (exception hierarchy, FastAPI integration), see [Error Handling in Python](/how-to/error-handling-patterns-in-python).

For TypeScript-specific patterns (exception hierarchy, Express/Next.js integration), see [Error Handling in TypeScript](/how-to/error-handling-patterns-in-typescript).

For Rust-specific patterns (RAII guard safety, `Error` enum matching, Axum integration), see [Error Handling in Rust](/how-to/error-handling-patterns-in-rust).

## Protocol error structure

The Python, Java, and TypeScript clients all expose structured error information when the server returns a protocol-level error.

::: code-group
```python [Python]
from runcycles import CyclesProtocolError

# Available attributes:
e.status           # HTTP status code (e.g. 409)
e.error_code       # Machine-readable error code (e.g. "BUDGET_EXCEEDED")
e.reason_code      # Reason code string
e.retry_after_ms   # Suggested retry delay in ms (or None)
e.request_id       # Server request ID
e.details          # Additional error details dict

# Convenience checks:
e.is_budget_exceeded()
e.is_overdraft_limit_exceeded()
e.is_debt_outstanding()
e.is_reservation_expired()
e.is_reservation_finalized()
e.is_idempotency_mismatch()
e.is_unit_mismatch()
e.is_retryable()
```
```java [Java]
public class CyclesProtocolException extends RuntimeException {
    ErrorCode getErrorCode();     // Machine-readable error code
    String getReasonCode();       // String error code
    int getHttpStatus();          // HTTP status from the server
    Integer getRetryAfterMs();    // Suggested retry delay (nullable)

    // Convenience checks
    boolean isBudgetExceeded();
    boolean isOverdraftLimitExceeded();
    boolean isDebtOutstanding();
    boolean isReservationExpired();
    boolean isReservationFinalized();
    boolean isIdempotencyMismatch();
    boolean isUnitMismatch();
}
```
```typescript [TypeScript]
import { CyclesProtocolError } from "runcycles";

// Available properties:
e.status;          // HTTP status code (e.g. 409)
e.errorCode;       // Machine-readable error code (e.g. "BUDGET_EXCEEDED")
e.reasonCode;      // Reason code string
e.retryAfterMs;    // Suggested retry delay in ms (or undefined)
e.requestId;       // Server request ID
e.details;         // Additional error details object

// Convenience checks:
e.isBudgetExceeded();
e.isOverdraftLimitExceeded();
e.isDebtOutstanding();
e.isReservationExpired();
e.isReservationFinalized();
e.isIdempotencyMismatch();
e.isUnitMismatch();
e.isRetryable();
```
:::

## Handling DENY decisions

When a reservation is denied, the decorated function / annotated method does not execute. An exception is thrown instead.

::: code-group
```python [Python]
from runcycles import cycles, BudgetExceededError, CyclesProtocolError

@cycles(estimate=1000)
def summarize(text: str) -> str:
    return call_llm(text)

try:
    result = summarize(text)
except BudgetExceededError:
    result = "Service temporarily unavailable due to budget limits."
except CyclesProtocolError as e:
    if e.retry_after_ms:
        schedule_retry(text, delay_ms=e.retry_after_ms)
        result = f"Request queued. Retrying in {e.retry_after_ms}ms."
    else:
        raise
```
```java [Java]
try {
    return llmService.summarize(text);
} catch (CyclesProtocolException e) {
    if (e.isBudgetExceeded() && e.getRetryAfterMs() != null) {
        scheduleRetry(text, e.getRetryAfterMs());
        return "Request queued. Retrying in " + e.getRetryAfterMs() + "ms.";
    }
    if (e.isBudgetExceeded()) {
        return fallbackSummary(text);
    }
    throw e;
}
```
```typescript [TypeScript]
import { withCycles, BudgetExceededError, CyclesProtocolError } from "runcycles";

const summarize = withCycles(
  { estimate: 1000, actionKind: "llm.completion", actionName: "gpt-4o", client },
  async (text: string) => callLlm(text),
);

try {
  result = await summarize(text);
} catch (err) {
  if (err instanceof BudgetExceededError) {
    result = "Service temporarily unavailable due to budget limits.";
  } else if (err instanceof CyclesProtocolError && err.retryAfterMs) {
    scheduleRetry(text, err.retryAfterMs);
    result = `Request queued. Retrying in ${err.retryAfterMs}ms.`;
  } else {
    throw err;
  }
}
```
:::

## Degradation patterns

::: code-group
```python [Python]
from runcycles import BudgetExceededError

try:
    result = premium_service.analyze(data)   # GPT-4o, high cost
except BudgetExceededError:
    result = basic_service.analyze(data)     # GPT-4o-mini, lower cost
```
```java [Java]
try {
    return premiumService.analyze(data);  // Uses GPT-4o, high cost
} catch (CyclesProtocolException e) {
    if (e.isBudgetExceeded()) {
        return basicService.analyze(data);  // Uses GPT-4o-mini, lower cost
    }
    throw e;
}
```
```typescript [TypeScript]
import { BudgetExceededError } from "runcycles";

try {
  result = await premiumService.analyze(data);   // GPT-4o, high cost
} catch (err) {
  if (err instanceof BudgetExceededError) {
    result = await basicService.analyze(data);   // GPT-4o-mini, lower cost
  } else {
    throw err;
  }
}
```
:::

## Handling debt and overdraft errors

### DebtOutstandingError / DEBT_OUTSTANDING

A scope has unpaid debt and no overdraft limit configured. New reservations are blocked until the debt is resolved or an overdraft limit is set.

::: code-group
```python [Python]
from runcycles import DebtOutstandingError

try:
    result = process(input_data)
except DebtOutstandingError:
    logger.warning("Scope has outstanding debt. Notifying operator.")
    alert_operator("Budget debt detected. Funding required.")
    result = "Service paused pending budget review."
```
```java [Java]
try {
    return service.process(input);
} catch (CyclesProtocolException e) {
    if (e.isDebtOutstanding()) {
        log.warn("Scope has outstanding debt. Notifying operator.");
        alertOperator("Budget debt detected. Funding required.");
        return "Service paused pending budget review.";
    }
    throw e;
}
```
```typescript [TypeScript]
import { DebtOutstandingError } from "runcycles";

try {
  result = await process(inputData);
} catch (err) {
  if (err instanceof DebtOutstandingError) {
    console.warn("Scope has outstanding debt. Notifying operator.");
    alertOperator("Budget debt detected. Funding required.");
    result = "Service paused pending budget review.";
  } else {
    throw err;
  }
}
```
:::

### OverdraftLimitExceededError / OVERDRAFT_LIMIT_EXCEEDED

The scope's debt has exceeded its overdraft limit.

::: code-group
```python [Python]
from runcycles import OverdraftLimitExceededError

try:
    result = process(input_data)
except OverdraftLimitExceededError:
    logger.error("Overdraft limit exceeded. Scope is blocked.")
    result = "Budget limit reached. Please contact support."
```
```java [Java]
try {
    return service.process(input);
} catch (CyclesProtocolException e) {
    if (e.isOverdraftLimitExceeded()) {
        log.error("Overdraft limit exceeded. Scope is blocked.");
        return "Budget limit reached. Please contact support.";
    }
    throw e;
}
```
```typescript [TypeScript]
import { OverdraftLimitExceededError } from "runcycles";

try {
  result = await process(inputData);
} catch (err) {
  if (err instanceof OverdraftLimitExceededError) {
    console.error("Overdraft limit exceeded. Scope is blocked.");
    result = "Budget limit reached. Please contact support.";
  } else {
    throw err;
  }
}
```
:::

## Handling expired reservations

If work outlives TTL plus grace, commit returns `RESERVATION_EXPIRED`. Current Python 0.5.2+, TypeScript 0.4.2+, Java/Spring 0.3.2+, and Rust 0.3.2+ lifecycle helpers recover known spend through `POST /v1/events`, reusing the original idempotency key. They persist the event-mode transition so a restart does not resume a commit that is already known to be impossible.

Do not create a new reservation after the work has completed, and do not release merely to restore budget. For low-level clients, construct a direct event from the original subject, action, actual amount, metrics, metadata, and key, and persist that recovery before sending it.

See [SDK Settlement Recovery and Durability](/protocol/sdk-settlement-recovery-and-durability) for the lifecycle-helper boundary and journal behavior.

## Catching all Cycles errors

::: code-group
```python [Python]
from runcycles import (
    BudgetExceededError,
    DebtOutstandingError,
    OverdraftLimitExceededError,
    CyclesProtocolError,
)

try:
    result = guarded_func()
except BudgetExceededError:
    result = fallback()
except DebtOutstandingError:
    alert_operator("Debt outstanding")
    result = "Service paused"
except OverdraftLimitExceededError:
    result = "Budget limit reached"
except CyclesProtocolError as e:
    if e.status == -1:
        # Transport failure — the request never got an HTTP response
        logger.error("Transport failure: %s", e)
    else:
        logger.error("Protocol error: %s (code=%s, status=%d)", e, e.error_code, e.status)
    raise

# Commit-time settlement recovery is durable once actual usage is known.
# Monitor retained/quarantined journal records and recovery warnings.
```
```java [Java]
try {
    return annotatedMethod();
} catch (CyclesProtocolException e) {
    if (e.isBudgetExceeded()) {
        return fallback();
    } else if (e.isDebtOutstanding()) {
        alertOperator("Debt outstanding");
        return "Service paused";
    } else if (e.isOverdraftLimitExceeded()) {
        return "Budget limit reached";
    } else {
        log.error("Protocol error: code={}, status={}", e.getReasonCode(), e.getHttpStatus());
        throw e;
    }
}
```
```typescript [TypeScript]
import {
  BudgetExceededError,
  DebtOutstandingError,
  OverdraftLimitExceededError,
  ReservationExpiredError,
  CyclesProtocolError,
} from "runcycles";

try {
  result = await guardedFunc();
} catch (err) {
  if (err instanceof BudgetExceededError) {
    result = await fallback();
  } else if (err instanceof DebtOutstandingError) {
    alertOperator("Debt outstanding");
    result = "Service paused";
  } else if (err instanceof OverdraftLimitExceededError) {
    result = "Budget limit reached";
  } else if (err instanceof ReservationExpiredError) {
    await recordAsEvent(data);
  } else if (err instanceof CyclesProtocolError) {
    if (err.status === -1) {
      // Transport failure — the request never got an HTTP response
      console.error(`Transport failure: ${err.message}`);
    } else {
      console.error(`Protocol error: ${err.message} (code=${err.errorCode}, status=${err.status})`);
    }
    throw err;
  } else {
    throw err;
  }
}
```
:::

## Web framework error handlers

::: code-group
```python [FastAPI]
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
from runcycles import CyclesProtocolError

app = FastAPI()

@app.exception_handler(CyclesProtocolError)
async def cycles_error_handler(request: Request, exc: CyclesProtocolError):
    if exc.is_budget_exceeded():
        retry_after = exc.retry_after_ms // 1000 if exc.retry_after_ms else 60
        return JSONResponse(
            status_code=429,
            content={"error": "budget_exceeded", "message": "Budget limit reached."},
            headers={"Retry-After": str(retry_after)},
        )

    if exc.is_debt_outstanding() or exc.is_overdraft_limit_exceeded():
        return JSONResponse(
            status_code=503,
            content={"error": "service_unavailable", "message": "Service paused due to budget constraints."},
        )

    return JSONResponse(
        status_code=500,
        content={"error": "internal_error", "message": "An unexpected error occurred."},
    )
```
```java [Spring]
@RestControllerAdvice
public class CyclesExceptionHandler {

    @ExceptionHandler(CyclesProtocolException.class)
    public ResponseEntity<Map<String, Object>> handleCyclesError(
            CyclesProtocolException e) {

        if (e.isBudgetExceeded()) {
            return ResponseEntity.status(429)
                .header("Retry-After",
                    String.valueOf(e.getRetryAfterMs() != null
                        ? e.getRetryAfterMs() / 1000 : 60))
                .body(Map.of(
                    "error", "budget_exceeded",
                    "message", "Budget limit reached. Please try again later."
                ));
        }

        if (e.isDebtOutstanding() || e.isOverdraftLimitExceeded()) {
            return ResponseEntity.status(503)
                .body(Map.of(
                    "error", "service_unavailable",
                    "message", "Service temporarily paused due to budget constraints."
                ));
        }

        return ResponseEntity.status(500)
            .body(Map.of(
                "error", "internal_error",
                "message", "An unexpected error occurred."
            ));
    }
}
```
```typescript [Express]
import type { Request, Response, NextFunction } from "express";
import { CyclesProtocolError } from "runcycles";

function cyclesErrorHandler(err: Error, req: Request, res: Response, next: NextFunction) {
  if (!(err instanceof CyclesProtocolError)) {
    return next(err);
  }

  if (err.isBudgetExceeded()) {
    const retryAfter = err.retryAfterMs ? Math.ceil(err.retryAfterMs / 1000) : 60;
    return res.status(429)
      .set("Retry-After", String(retryAfter))
      .json({ error: "budget_exceeded", message: "Budget limit reached." });
  }

  if (err.isDebtOutstanding() || err.isOverdraftLimitExceeded()) {
    return res.status(503)
      .json({ error: "service_unavailable", message: "Service paused due to budget constraints." });
  }

  return res.status(500)
    .json({ error: "internal_error", message: "An unexpected error occurred." });
}
```
```typescript [Next.js]
import { BudgetExceededError } from "runcycles";

export async function POST(req: Request) {
  try {
    const result = await handleChat(req);
    return new Response(result);
  } catch (err) {
    if (err instanceof BudgetExceededError) {
      return new Response(
        JSON.stringify({ error: "budget_exceeded", message: "Budget limit reached." }),
        { status: 402, headers: { "Content-Type": "application/json" } },
      );
    }
    throw err;
  }
}
```
:::

## Programmatic client error handling

When using the client directly, errors come as response status codes rather than exceptions.

::: code-group
```python [Python]
from runcycles import CyclesClient

with CyclesClient(config) as client:
    response = client.create_reservation(request)

    if response.is_success:
        reservation_id = response.get_body_attribute("reservation_id")
        # Proceed with work
    elif response.is_server_error:
        # Server error — retry with backoff
        logger.warning("Cycles server error: %s", response.error_message)
    elif response.is_transport_error:
        # Network failure — retry with backoff
        logger.warning("Transport error: %s", response.error_message)
    else:
        # Client error (4xx) — do not retry
        # 409 = budget exceeded, debt outstanding, overdraft limit exceeded
        # 400 = invalid request, unit mismatch
        # 410 = reservation expired
        logger.error(
            "Cycles client error: status=%d, error=%s",
            response.status, response.error_message,
        )
```
```java [Java]
CyclesResponse<Map<String, Object>> response = cyclesClient.createReservation(request);

if (response.is2xx()) {
    // For non-dry-run reservations, a 2xx response means ALLOW or ALLOW_WITH_CAPS.
    // Insufficient budget returns 409 (handled below by the else branch).
    // Proceed with work
} else if (response.is5xx() || response.isTransportError()) {
    // Server error or network issue — retry
    log.warn("Cycles server error: {}", response.getErrorMessage());
    return retryOrFallback();
} else {
    // Client error (4xx) — do not retry
    // 409 = budget exceeded, debt outstanding, overdraft limit exceeded
    // 400 = invalid request, unit mismatch
    // 410 = reservation expired
    log.error("Cycles client error: status={}, error={}",
        response.getStatus(), response.getErrorMessage());
    throw new RuntimeException("Cycles request failed: " + response.getErrorMessage());
}
```
```typescript [TypeScript]
const response = await client.createReservation(request);

if (response.isSuccess) {
  const reservationId = response.getBodyAttribute("reservation_id") as string;
  // Proceed with work
} else if (response.isServerError || response.isTransportError) {
  // Server error or network failure — retry with backoff
  console.warn(`Cycles server error: ${response.errorMessage}`);
} else {
  // Client error (4xx) — do not retry
  // 409 = budget exceeded, debt outstanding, overdraft limit exceeded
  // 400 = invalid request, unit mismatch
  // 410 = reservation expired
  console.error(`Cycles client error: status=${response.status}, error=${response.errorMessage}`);
}
```
:::

## Transient vs non-transient errors

| Error | Retryable? | Action |
|---|---|---|
| `BUDGET_EXCEEDED` (409) | Maybe | Budget may free up after other reservations commit. Retry with backoff or degrade. |
| `DEBT_OUTSTANDING` (409) | Wait | Requires operator to fund the scope or configure an overdraft limit. Retry after funding. |
| `OVERDRAFT_LIMIT_EXCEEDED` (409) | Wait | Requires operator intervention. |
| `RESERVATION_EXPIRED` (410) | No | Lifecycle helpers recover known spend as a same-key event; low-level callers must persist and perform that fallback. |
| `RESERVATION_FINALIZED` (409) | No | Reservation already settled. No action needed. |
| `IDEMPOTENCY_MISMATCH` (409) | No | Fix the idempotency key or payload. |
| `UNIT_MISMATCH` (400) | No | Fix the unit in your request. Inspect `details.expected_units` to see which units are funded at the scope. |
| `INVALID_REQUEST` (400) | No | Fix the request payload. |
| `UNAUTHORIZED` (401) | No | Fix the API key. |
| `FORBIDDEN` (403) | No | Fix the tenant configuration. |
| `NOT_FOUND` (404) | No / Wait | Two cases, distinguished by the `message` field: missing reservation (`"Reservation not found: ..."` — check the reservation ID) or missing budget (`"Budget not found for provided scope: ..."` — operator must create a budget via the admin API). |
| `INTERNAL_ERROR` (500) | Yes | Retry with exponential backoff. |
| Transport error | Yes | Retry with exponential backoff. |

In Python and TypeScript, use `e.is_retryable()` / `e.isRetryable()` to check programmatically — it returns `true` for `INTERNAL_ERROR`, `UNKNOWN`, and any 5xx status.

## Error handling checklist

1. **Always catch protocol errors** (`CyclesProtocolError` / `CyclesProtocolException`) at the boundary where user-facing behavior is determined
2. **Use specific subclasses** (`BudgetExceededError`, `DebtOutstandingError`, etc.) for precise handling in Python
3. **Check `retry_after_ms`** before implementing your own retry delay
4. **Distinguish between DENY and server errors** — DENY means the system is working correctly, server errors mean something is wrong
5. **Log `error_code` and `status`** for debugging
6. **Never swallow errors silently** — at minimum, log them
7. **Handle `RESERVATION_EXPIRED` correctly** — lifecycle helpers recover known usage automatically; low-level clients must persist and send the same-key event fallback
8. **Register a global exception handler** in web frameworks for consistent API error responses
9. **Avoid nested budget guards** — Spring throws `IllegalStateException`; TypeScript/Python silently double-count. Place `@Cycles` / `withCycles` / `@cycles` at the outermost call site only (see [Troubleshooting](/how-to/troubleshooting-and-faq#spring-boot-illegalstateexception-nested-cycles))

## Next steps

- [Error Handling in TypeScript](/how-to/error-handling-patterns-in-typescript) — TypeScript exception hierarchy, Express/Next.js patterns
- [Error Handling in Python](/how-to/error-handling-patterns-in-python) — Python exception hierarchy, transport errors, and FastAPI patterns
- [Error Codes and Error Handling](/protocol/error-codes-and-error-handling-in-cycles) — protocol error code reference
- [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) — strategies for handling budget constraints
- [Using the Client Programmatically](/how-to/using-the-cycles-client-programmatically) — direct client usage patterns



# Error Handling Patterns in Python

This guide covers practical patterns for handling Cycles errors in Python applications — both with the `@cycles` decorator and with the programmatic `CyclesClient`.

::: tip Also available
See [Error Handling in TypeScript](/how-to/error-handling-patterns-in-typescript) or [Rust](/how-to/error-handling-patterns-in-rust) for equivalent patterns, or the [general Error Handling Patterns](/how-to/error-handling-patterns-in-cycles-client-code) for language-agnostic concepts.
:::

## Exception hierarchy

The `runcycles` package provides a typed exception hierarchy:

```
CyclesError (base)
├── CyclesProtocolError (server returned a protocol-level error)
│   ├── BudgetExceededError
│   ├── OverdraftLimitExceededError
│   ├── DebtOutstandingError
│   ├── ReservationExpiredError
│   └── ReservationFinalizedError
└── CyclesTransportError (exported for user code; not raised by the SDK)
```

`CyclesTransportError` is exported but never raised by the SDK itself — transport failures surface as `status == -1` instead. See [Transport errors](#transport-errors) below.

## CyclesProtocolError

When the `@cycles` decorator encounters a DENY decision or a protocol error, it raises `CyclesProtocolError`. A specific subclass (`BudgetExceededError`, etc.) is raised only when the server returns a matching HTTP error code; a 200 response with `decision=DENY` raises plain `CyclesProtocolError` with `reason_code` set:

```python
from runcycles import CyclesProtocolError

# Available attributes:
e.status           # HTTP status code (e.g. 409)
e.error_code       # Machine-readable error code (e.g. "BUDGET_EXCEEDED")
e.reason_code      # Reason code string
e.retry_after_ms   # Suggested retry delay in ms (or None)
e.request_id       # Server request ID
e.details          # Additional error details dict

# Convenience checks:
e.is_budget_exceeded()
e.is_overdraft_limit_exceeded()
e.is_debt_outstanding()
e.is_reservation_expired()
e.is_reservation_finalized()
e.is_idempotency_mismatch()
e.is_unit_mismatch()
e.is_retryable()
```

## Handling DENY decisions

When a reservation is denied, the decorated function does not execute. An exception is raised instead.

### Basic catch

```python
from runcycles import cycles, BudgetExceededError

@cycles(estimate=1000)
def summarize(text: str) -> str:
    return call_llm(text)

try:
    result = summarize(text)
except BudgetExceededError:
    result = "Service temporarily unavailable due to budget limits."
```

### With retry delay

The server may include a `retry_after_ms` hint suggesting when budget might become available:

```python
from runcycles import CyclesProtocolError

try:
    result = summarize(text)
except CyclesProtocolError as e:
    if e.is_budget_exceeded() and e.retry_after_ms:
        schedule_retry(text, delay_ms=e.retry_after_ms)
        result = f"Request queued. Retrying in {e.retry_after_ms}ms."
    elif e.is_budget_exceeded():
        result = fallback_summary(text)
    else:
        raise
```

### Degradation patterns

```python
from runcycles import BudgetExceededError

try:
    result = premium_service.analyze(data)   # GPT-4o, high cost
except BudgetExceededError:
    result = basic_service.analyze(data)     # GPT-4o-mini, lower cost
```

## Handling debt and overdraft errors

### DebtOutstandingError

A scope has unpaid debt and no overdraft limit configured. New reservations are blocked until the debt is resolved or an overdraft limit is set.

```python
from runcycles import DebtOutstandingError

try:
    result = process(input_data)
except DebtOutstandingError:
    logger.warning("Scope has outstanding debt. Notifying operator.")
    alert_operator("Budget debt detected. Funding required.")
    result = "Service paused pending budget review."
```

### OverdraftLimitExceededError

The scope's debt has exceeded its overdraft limit.

```python
from runcycles import OverdraftLimitExceededError

try:
    result = process(input_data)
except OverdraftLimitExceededError:
    logger.error("Overdraft limit exceeded. Scope is blocked.")
    result = "Budget limit reached. Please contact support."
```

## Handling expired reservations

If a function takes longer than the reservation TTL plus grace period, the commit fails with `RESERVATION_EXPIRED`. The decorator handles heartbeat extensions automatically, but network issues can prevent extensions.

In `runcycles` 0.5.2+, the decorator and streaming lifecycle helpers persist known actual usage before commit. A commit-time `RESERVATION_EXPIRED` switches the durable record to event mode and records the spend through `POST /v1/events` with the original idempotency key. If event recovery remains ambiguous or unavailable, the record stays journaled for later replay.

`ReservationExpiredError` can still surface from a low-level operation. Direct `commit_reservation()` calls do not automatically create or persist the event fallback; callers using the low-level client must provide equivalent recovery.

## Catching all Cycles errors

```python
from runcycles import (
    BudgetExceededError,
    DebtOutstandingError,
    OverdraftLimitExceededError,
    CyclesProtocolError,
    CyclesError,
)

try:
    result = guarded_func()
except BudgetExceededError:
    result = fallback()
except DebtOutstandingError:
    alert_operator("Debt outstanding")
    result = "Service paused"
except OverdraftLimitExceededError:
    result = "Budget limit reached"
except CyclesProtocolError as e:
    if e.status == -1:
        # Network-level failure at reserve time — retry with backoff
        logger.error("Transport error: %s", e)
    else:
        # Any other protocol error
        logger.error("Protocol error: %s (code=%s, status=%d)", e, e.error_code, e.status)
    raise
```

## Programmatic client error handling

When using `CyclesClient` directly, errors come as response status codes rather than exceptions:

```python
from runcycles import CyclesClient, ReservationCreateRequest

with CyclesClient(config) as client:
    response = client.create_reservation(request)

    if response.is_success:
        reservation_id = response.get_body_attribute("reservation_id")
        # Proceed with work
    elif response.is_server_error:
        # Server error — retry with backoff
        logger.warning("Cycles server error: %s", response.error_message)
    elif response.is_transport_error:
        # Network failure (status == -1) — retry with backoff
        logger.warning("Transport error: %s", response.error_message)
    else:
        # Client error (4xx) — do not retry
        # 409 = budget exceeded, debt outstanding, overdraft limit exceeded
        # 400 = invalid request, unit mismatch
        # 410 = reservation expired
        logger.error(
            "Cycles client error: status=%d, error=%s",
            response.status, response.error_message,
        )
```

## Transport errors

When the HTTP request itself fails (DNS resolution, connection refused, timeout), how it surfaces depends on the API:

- **Decorator:** a transport failure at reserve time raises `CyclesProtocolError` with `status == -1` and `error_code=None`. Once actual usage is known, transport failures at commit time retain a durable same-key record and retry in the background.
- **Programmatic client:** calls never raise for transport failures — they return a `CyclesResponse` with `is_transport_error == True` and `status == -1`.

```python
from runcycles import CyclesProtocolError

try:
    result = guarded_func()
except CyclesProtocolError as e:
    if e.status == -1:
        logger.error("Network error reaching Cycles: %s", e)
        # Retry or degrade
    else:
        raise
```

The `CyclesTransportError` class is exported for use in your own code (e.g. wrapping transport-level failures in higher-level integrations), but the SDK itself never raises it.

## FastAPI / Starlette error handler

For web applications, register a global exception handler:

```python
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
from runcycles import CyclesProtocolError

app = FastAPI()

@app.exception_handler(CyclesProtocolError)
async def cycles_error_handler(request: Request, exc: CyclesProtocolError):
    if exc.is_budget_exceeded():
        retry_after = exc.retry_after_ms // 1000 if exc.retry_after_ms else 60
        return JSONResponse(
            status_code=429,
            content={"error": "budget_exceeded", "message": "Budget limit reached."},
            headers={"Retry-After": str(retry_after)},
        )

    if exc.is_debt_outstanding() or exc.is_overdraft_limit_exceeded():
        return JSONResponse(
            status_code=503,
            content={"error": "service_unavailable", "message": "Service paused due to budget constraints."},
        )

    return JSONResponse(
        status_code=500,
        content={"error": "internal_error", "message": "An unexpected error occurred."},
    )
```

## Transient vs non-transient errors

| Error | Retryable? | Action |
|---|---|---|
| `BUDGET_EXCEEDED` (409) | Maybe | Budget may free up after other reservations commit. Retry with backoff or degrade. |
| `BUDGET_FROZEN` (409) | Wait | Budget scope frozen by an operator. Retry after it is unfrozen. |
| `BUDGET_CLOSED` (409) | No | Budget scope is permanently closed. Route spend elsewhere. |
| `DEBT_OUTSTANDING` (409) | Wait | Requires operator to fund the scope or configure an overdraft limit. Retry after funding. |
| `OVERDRAFT_LIMIT_EXCEEDED` (409) | Wait | Requires operator intervention. |
| `MAX_EXTENSIONS_EXCEEDED` (409) | No | Tenant's `max_reservation_extensions` limit reached. Commit or release; use a longer initial `ttl_ms`. |
| `RESERVATION_EXPIRED` (410) | No | Lifecycle helpers recover known spend as a same-key event; low-level callers must persist and perform that fallback. |
| `RESERVATION_FINALIZED` (409) | No | Reservation already settled. No action needed. |
| `IDEMPOTENCY_MISMATCH` (409) | No | Fix the idempotency key or payload. |
| `UNIT_MISMATCH` (400) | No | Fix the unit in your request. Inspect `details.expected_units` to see which units are funded at the scope. |
| `INVALID_REQUEST` (400) | No | Fix the request payload. |
| `NOT_FOUND` (404) | No / Wait | Two cases, distinguished by the `message` field: missing reservation (`"Reservation not found: ..."` — check the reservation ID) or missing budget (`"Budget not found for provided scope: ..."` — operator must create a budget via the admin API). |
| `INTERNAL_ERROR` (500) | Yes | Retry with exponential backoff. |
| Transport error | Yes | Retry with exponential backoff. |

Use `e.is_retryable()` to check programmatically — it returns `True` for `INTERNAL_ERROR`, `UNKNOWN`, and any 5xx status.

## Error handling checklist

1. **Always catch `CyclesProtocolError`** at the boundary where user-facing behavior is determined
2. **Use specific subclasses** (`BudgetExceededError`, `DebtOutstandingError`, etc.) for precise handling
3. **Check `retry_after_ms`** before implementing your own retry delay
4. **Distinguish between DENY and server errors** — DENY means the system is working correctly, server errors mean something is wrong
5. **Log `error_code` and `status`** for debugging
6. **Never swallow errors silently** — at minimum, log them
7. **Monitor durable recovery** — alert on journal I/O failures, quarantined records, authentication failures, retry exhaustion, and expired-commit event fallback failures
8. **Register a global exception handler** in web frameworks for consistent API error responses

## Next steps

- [Getting Started with the Python Client](/quickstart/getting-started-with-the-python-client) — decorator and client setup
- [Error Codes and Error Handling](/protocol/error-codes-and-error-handling-in-cycles) — protocol error code reference
- [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) — strategies for handling budget constraints



# Rust Error Handling for AI Agent Budget Failures

When your Rust async AI agent hits a budget limit, runs out of tokens, or sees a transient network failure mid-reservation — what should it do? This guide covers idiomatic Rust patterns for handling Cycles errors in async applications: retries, RAII-driven cleanup, graceful degradation, and the `BUDGET_EXCEEDED` / `RESERVATION_EXPIRED` decision points. Patterns work with `with_cycles()`, `ReservationGuard`, and the programmatic `CyclesClient`.

::: tip Also available
See [Error Handling in Python](/how-to/error-handling-patterns-in-python) or [TypeScript](/how-to/error-handling-patterns-in-typescript) for equivalent patterns, or the [general Error Handling Patterns](/how-to/error-handling-patterns-in-cycles-client-code) for language-agnostic concepts.
:::

## Error enum

The `runcycles` crate uses a single `Error` enum (not a trait-object hierarchy):

```
Error
├── Transport(reqwest::Error)        — network failure, timeout, DNS
├── Api { status, code, message, … } — server returned an error response
├── BudgetExceeded { message, … }    — budget insufficient (HTTP 409)
├── CommitPending { reservation_id, last_error } — settlement is durably queued
├── CommitRecoveryFailed { … }       — fallback failed without durable journaling
├── Deserialization(serde_json::Error) — response body parse failure
├── Config(String)                    — invalid client configuration
└── Validation(String)                — invalid request (caught before sending)
```

## Error methods

Every `Error` variant exposes convenience methods:

```rust
use runcycles::Error;

fn handle(err: &Error) {
    err.is_retryable()       // true for Transport, 5xx, BudgetExceeded with retry_after,
                             // or Api with a retryable code (INTERNAL_ERROR / unrecognized)
    err.is_budget_exceeded() // true for BudgetExceeded or Api with code BudgetExceeded
    err.retry_after()        // Option<Duration> — server-suggested delay
    err.request_id()         // Option<&str> — server-assigned request ID
    err.error_code()         // Option<ErrorCode> — parsed error code
}
```

## Handling DENY decisions

When a reservation is denied, `with_cycles()` returns `Err(Error::BudgetExceeded { .. })`. The guarded closure does not execute.

### Basic catch

```rust
use runcycles::{with_cycles, WithCyclesConfig, Error, models::Amount};

let config = WithCyclesConfig::new(Amount::tokens(1000))
    .action("llm.completion", "gpt-4o");

let result = with_cycles(&client, config, |ctx| async move {
    let response = call_llm(&prompt).await?;
    let actual = Amount::tokens(42);
    Ok((response, actual))
}).await;

match result {
    Ok(response) => println!("Success: {response}"),
    Err(Error::BudgetExceeded { message, .. }) => {
        println!("Budget exceeded: {message}");
        // Fall back to cheaper model or cached response
    }
    Err(e) => return Err(e.into()),
}
```

The closure receives a `GuardContext` (with `decision`, `caps`, `reservation_id`, `affected_scopes`) and must return `Result<(T, Amount), Box<dyn std::error::Error + Send + Sync>>` (note: not `runcycles::Error`) — the value plus the actual cost for commit.

### With retry delay

The server may include a suggested retry delay:

```rust
match result {
    Err(Error::BudgetExceeded { retry_after, .. }) if retry_after.is_some() => {
        let delay = retry_after.unwrap();
        println!("Budget exceeded. Retrying in {delay:?}");
        tokio::time::sleep(delay).await;
        // Retry the operation
    }
    Err(Error::BudgetExceeded { .. }) => {
        // No retry hint — degrade immediately
        fallback_response()
    }
    _ => { /* ... */ }
}
```

### Degradation pattern

```rust
let premium = WithCyclesConfig::new(Amount::tokens(2000))
    .action("llm.completion", "gpt-4o");

let result = with_cycles(&client, premium, |_ctx| async move {
    let r = call_llm_gpt4o(&prompt).await?;
    Ok((r, Amount::tokens(1800)))
}).await;

let response = match result {
    Ok(r) => r,
    Err(Error::BudgetExceeded { .. }) => {
        // Try cheaper model with lower estimate
        let budget = WithCyclesConfig::new(Amount::tokens(500))
            .action("llm.completion", "gpt-4o-mini");
        with_cycles(&client, budget, |_ctx| async move {
            let r = call_llm_gpt4o_mini(&prompt).await?;
            Ok((r, Amount::tokens(300)))
        }).await?
    }
    Err(e) => return Err(e.into()),
};
```

## ReservationGuard RAII safety

The `ReservationGuard` provides compile-time and runtime safety that Python and TypeScript cannot:

### Compile-time: no double-commit

`commit()` and `release()` take `self` by value, consuming the guard. You cannot call either twice:

```rust
let guard = client.reserve(request).await?;

guard.commit(commit_req).await?;
// guard.commit(another_req).await?;  // ← Compile error: use of moved value
```

### Runtime: auto-release on drop

If a guard is dropped without `commit()` or `release()` (panic, early `?` return, scope exit), it attempts a best-effort release:

```rust
async fn process(client: &CyclesClient) -> Result<String, Error> {
    let guard = client.reserve(request).await?;

    let result = do_work().await?;  // ← If this fails, guard is dropped

    // Guard auto-releases via Drop — budget returns to pool
    // No leaked reservation, no manual cleanup needed

    guard.commit(commit_req).await?;
    Ok(result)
}
```

### Inspecting caps before execution

```rust
let guard = client.reserve(request).await?;

if guard.is_capped() {
    if let Some(caps) = guard.caps() {
        if let Some(max_tokens) = caps.max_tokens {
            // Reduce output length
            prompt_config.max_tokens = max_tokens as usize;
        }
        if let Some(ref denylist) = caps.tool_denylist {
            // Remove denied tools
            available_tools.retain(|t| !denylist.contains(&t.name));
        }
    }
}

let result = execute_with_config(&prompt_config).await?;
guard.commit(commit_req).await?;
```

## Handling API errors

`Error::Api` covers all non-budget server errors:

`ReservationGuard::commit()` handles an expired commit itself: it switches the durable record to event mode and calls `POST /v1/events`. The HTTP 410 arm below therefore applies to low-level client calls, where the application owns persistence and event construction.

```rust
match result {
    Err(Error::Api { status, code, message, request_id, .. }) => {
        match status {
            409 => {
                // Note: the client converts 409s carrying BUDGET_EXCEEDED,
                // DEBT_OUTSTANDING, or OVERDRAFT_LIMIT_EXCEEDED into
                // Error::BudgetExceeded before they reach this arm — see
                // "Handling budget denials" above. Only non-budget 409s
                // (e.g. RESERVATION_FINALIZED) arrive as Error::Api.
                match code {
                    Some(ErrorCode::ReservationFinalized) => {
                        tracing::warn!("Reservation already finalized — no action needed");
                    }
                    _ => tracing::error!("API error 409: {message}"),
                }
            }
            410 => {
                // Reservation expired — work may have already run
                tracing::warn!("Reservation expired. Recording as event.");
                record_as_event(&client, actual_cost).await?;
            }
            400 => {
                // Invalid request — do not retry
                tracing::error!("Invalid request: {message}");
            }
            500.. => {
                // Server error — retry
                tracing::error!(
                    request_id = request_id.as_deref().unwrap_or("unknown"),
                    "Server error {status}: {message}"
                );
            }
            _ => tracing::error!("Unexpected status {status}: {message}"),
        }
    }
    _ => { /* ... */ }
}
```

## Handling transport errors

Network failures, timeouts, DNS resolution:

```rust
match result {
    Err(Error::Transport(ref reqwest_err)) => {
        if reqwest_err.is_timeout() {
            tracing::warn!("Cycles request timed out — retrying");
        } else if reqwest_err.is_connect() {
            tracing::error!("Cannot reach Cycles server — check network");
        } else {
            tracing::error!("Transport error: {reqwest_err}");
        }
        // All transport errors are retryable
    }
    _ => { /* ... */ }
}
```

## Catching all Cycles errors

```rust
use runcycles::{with_cycles, WithCyclesConfig, Error, models::{Amount, ErrorCode}};

let config = WithCyclesConfig::new(Amount::tokens(1000))
    .action("llm.completion", "gpt-4o");

let result = with_cycles(&client, config, |ctx| async move {
    let value = do_work().await?;
    Ok((value, Amount::tokens(800)))
}).await;

match result {
    Ok(value) => Ok(value),

    // Budget denied — degrade. This also covers 409s the client
    // collapses into BudgetExceeded (DEBT_OUTSTANDING,
    // OVERDRAFT_LIMIT_EXCEEDED); inspect `message` if you need
    // to tell them apart.
    Err(Error::BudgetExceeded { message, .. }) => {
        tracing::info!("Budget exceeded: {message}");
        Ok(fallback_value())
    }

    // Server/protocol error — check retryability
    Err(e @ Error::Api { .. }) if e.is_retryable() => {
        tracing::warn!("Retryable API error: {e}");
        Err(e)
    }

    // Network failure — retry
    Err(e @ Error::Transport(_)) => {
        tracing::warn!("Transport error: {e}");
        Err(e)
    }

    // Non-retryable — fail
    Err(e) => {
        tracing::error!("Non-retryable error: {e}");
        Err(e)
    }
}
```

## Axum error handler

For Axum web applications, convert Cycles errors to HTTP responses:

```rust
use axum::response::{IntoResponse, Response};
use axum::http::StatusCode;
use axum::Json;
use runcycles::Error;
use serde_json::json;

impl IntoResponse for AppError {
    fn into_response(self) -> Response {
        match &self.0 {
            Error::BudgetExceeded { retry_after, .. } => {
                let retry_secs = retry_after
                    .map(|d| d.as_secs().to_string())
                    .unwrap_or_else(|| "60".to_string());

                (
                    StatusCode::TOO_MANY_REQUESTS,
                    [("Retry-After", retry_secs)],
                    Json(json!({"error": "budget_exceeded", "message": "Budget limit reached."})),
                ).into_response()
            }

            // Debt and overdraft 409s arrive as Error::BudgetExceeded
            // (the client collapses those codes), so they are handled by
            // the arm above. Remaining Api errors:
            _ => {
                (
                    StatusCode::INTERNAL_SERVER_ERROR,
                    Json(json!({"error": "internal_error", "message": "An unexpected error occurred."})),
                ).into_response()
            }
        }
    }
}

struct AppError(Error);
impl From<Error> for AppError {
    fn from(e: Error) -> Self { Self(e) }
}
```

## Transient vs non-transient errors

| Error | Retryable? | Action |
|---|---|---|
| `BudgetExceeded` (409) | Maybe | Budget may free up. Check `retry_after`. Retry or degrade. |
| `BudgetExceeded` carrying a debt message (409 `DEBT_OUTSTANDING`) | Wait | Requires operator to fund the scope; the client collapses this code into `BudgetExceeded`. |
| `BudgetExceeded` carrying an overdraft message (409 `OVERDRAFT_LIMIT_EXCEEDED`) | Wait | Requires operator intervention; also collapsed into `BudgetExceeded`. |
| `CommitPending` | Replay queued | Do not compensate with a new key; drain or allow startup replay. |
| `Api` with `ReservationExpired` (410) | No | Guard commit recovers automatically; low-level callers persist and record the known spend as a same-key event. |
| `Api` with `ReservationFinalized` (409) | No | Already settled. No action needed. |
| `Api` with 5xx | Yes | Retry with exponential backoff. |
| `Transport` | Yes | Retry with exponential backoff. |
| `Deserialization` | No | Bug — report. |
| `Config` | No | Fix configuration before startup. |
| `Validation` | No | Fix request parameters. |

Use `error.is_retryable()` to check programmatically.

## Rust-specific advantages

| Feature | Rust | Python / TypeScript |
|---|---|---|
| Double-commit prevention | **Compile-time** (guard consumed by value) | Runtime exception |
| Auto-release on failure | **RAII Drop** (works on panic, `?`, scope exit) | `try/finally` or `async with` |
| Error exhaustiveness | **`match` requires all variants** | Catch-all or unhandled |
| Retryability check | `error.is_retryable()` built-in | `error.is_retryable()` |
| Heartbeat | Automatic via `tokio::spawn` | Automatic via background task |

## Error handling checklist

1. **Always match on `Error::BudgetExceeded`** at the boundary where user-facing behavior is determined
2. **Use `guard.is_capped()`** to inspect ALLOW_WITH_CAPS decisions before executing
3. **Let RAII handle cleanup** — don't manually release in every error path; the `Drop` impl does it
4. **Check `error.is_retryable()`** before implementing retry logic
5. **Check `error.retry_after()`** before choosing your own delay
6. **Log `error.request_id()`** for debugging server-side issues
7. **Treat `Error::CommitPending` as queued, not rejected** — never compensate with a different key
8. **Handle low-level reservation expiry** by durably recording known usage as a same-key event; guarded commit does this automatically
9. **Implement `IntoResponse`** in web frameworks for consistent API error responses

## Next steps

- [Getting Started with the Rust Client](/quickstart/getting-started-with-the-rust-client) — `with_cycles()`, `ReservationGuard`, and `CyclesClient` setup
- [Error Codes and Error Handling](/protocol/error-codes-and-error-handling-in-cycles) — protocol error code reference
- [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) — strategies for handling budget constraints
- [SDK Settlement Recovery and Durability](/protocol/sdk-settlement-recovery-and-durability) — journal, replay, expiry fallback, and guarantee boundary
- [How to Add Budget and Action Guardrails to Rust AI Agents](/blog/how-to-add-budget-and-action-guardrails-to-rust-ai-agents-with-cycles) — end-to-end Rust agent example



# Error Handling Patterns in TypeScript

This guide covers practical patterns for handling Cycles errors in TypeScript applications — with `withCycles`, `reserveForStream`, and the programmatic `CyclesClient`.

::: tip Also available
See [Error Handling in Python](/how-to/error-handling-patterns-in-python) or [Rust](/how-to/error-handling-patterns-in-rust) for equivalent patterns, or the [general Error Handling Patterns](/how-to/error-handling-patterns-in-cycles-client-code) for language-agnostic concepts.
:::

## Exception hierarchy

The `runcycles` package provides a typed exception hierarchy:

```
CyclesError (base)
├── CyclesProtocolError (server returned a protocol-level error)
│   ├── BudgetExceededError
│   ├── OverdraftLimitExceededError
│   ├── DebtOutstandingError
│   ├── ReservationExpiredError
│   └── ReservationFinalizedError
└── CyclesTransportError (exported for user code; not thrown by the SDK)
```

## CyclesProtocolError

When `withCycles` or `reserveForStream` encounters a DENY decision or a protocol error, it throws `CyclesProtocolError` (or a specific subclass):

```typescript
import { CyclesProtocolError } from "runcycles";

// Available properties:
e.status;          // HTTP status code (e.g. 409)
e.errorCode;       // Machine-readable error code (e.g. "BUDGET_EXCEEDED")
e.reasonCode;      // Reason code string
e.retryAfterMs;    // Suggested retry delay in ms (or undefined)
e.requestId;       // Server request ID
e.details;         // Additional error details (Record<string, unknown>)

// Convenience checks:
e.isBudgetExceeded();
e.isOverdraftLimitExceeded();
e.isDebtOutstanding();
e.isReservationExpired();
e.isReservationFinalized();
e.isIdempotencyMismatch();
e.isUnitMismatch();
e.isRetryable();           // true for INTERNAL_ERROR, UNKNOWN, or 5xx status
```

## Transport failures (status -1)

Transport failures (DNS failure, timeout, connection refused) do not surface as a distinct exception class. The SDK never throws `CyclesTransportError` itself — the class is exported for use in your own code. Instead:

- **`withCycles` / `reserveForStream`** throw `CyclesProtocolError` with `status === -1` and `errorCode` `undefined`.
- **Programmatic `CyclesClient` calls** never throw on transport failure — they return a `CyclesResponse` with `isTransportError` set and `status` of `-1`.

Detect transport failures by checking the status:

```typescript
import { CyclesProtocolError } from "runcycles";

try {
  result = await guardedFunc();
} catch (err) {
  if (err instanceof CyclesProtocolError && err.status === -1) {
    // Network-level failure — the server was never reached
    console.error(`Transport error: ${err.message}`);
  }
}
```

For the programmatic client:

```typescript
const response = await client.createReservation(body);
if (response.isTransportError) {
  console.error(`Transport error: ${response.errorMessage}`);
  console.error(`Cause: ${response.transportError}`);
}
```

## Catching errors from withCycles

`withCycles` wraps a function with the full reserve → execute → commit lifecycle. If the reservation is denied, it throws before your function runs:

In `runcycles` 0.4.2+, known actual usage is journaled before the first commit request. Ambiguous settlement, authentication failure, and retry exhaustion retain the same-key record for replay; an expired commit switches to a durable `/v1/events` fallback. These commit-time recovery paths are reported operationally rather than converted into a second execution of the guarded function.

```typescript
import { withCycles, BudgetExceededError, CyclesProtocolError } from "runcycles";

const summarize = withCycles(
  { estimate: 1000, actionKind: "llm.completion", actionName: "gpt-4o", client },
  async (text: string) => callLlm(text),
);

try {
  const result = await summarize(text);
} catch (err) {
  if (err instanceof BudgetExceededError) {
    // Budget exhausted — degrade or queue
    return fallbackSummary(text);
  } else if (err instanceof CyclesProtocolError) {
    // Other protocol error
    if (err.retryAfterMs) {
      scheduleRetry(text, err.retryAfterMs);
      return `Request queued. Retrying in ${err.retryAfterMs}ms.`;
    }
    throw err;
  } else {
    throw err;
  }
}
```

## Catching errors from reserveForStream

`reserveForStream` throws on reservation failure. After a successful reservation, you must handle errors from the stream itself and release the handle:

```typescript
import { reserveForStream, BudgetExceededError } from "runcycles";

let handle;
try {
  handle = await reserveForStream({
    client,
    estimate: estimatedCost,
    actionKind: "llm.completion",
    actionName: "gpt-4o",
  });
} catch (err) {
  if (err instanceof BudgetExceededError) {
    console.error("Budget exhausted:", err.message);
    return;
  }
  throw err;
}

// Stream with cleanup on failure
try {
  const stream = await openai.chat.completions.create({ model: "gpt-4o", messages, stream: true });
  // ... process stream ...
  await handle.commit(actualCost, metrics);
} catch (err) {
  await handle.release("stream_error");
  throw err;
}
```

## Express middleware error handling

Register a global error handler that catches Cycles errors and returns appropriate HTTP responses:

```typescript
import type { Request, Response, NextFunction } from "express";
import { CyclesProtocolError, BudgetExceededError } from "runcycles";

function cyclesErrorHandler(err: Error, req: Request, res: Response, next: NextFunction) {
  if (!(err instanceof CyclesProtocolError)) {
    return next(err);
  }

  if (err.isBudgetExceeded()) {
    const retryAfter = err.retryAfterMs ? Math.ceil(err.retryAfterMs / 1000) : 60;
    return res.status(429)
      .set("Retry-After", String(retryAfter))
      .json({ error: "budget_exceeded", message: "Budget limit reached." });
  }

  if (err.isDebtOutstanding() || err.isOverdraftLimitExceeded()) {
    return res.status(503)
      .json({ error: "service_unavailable", message: "Service paused due to budget constraints." });
  }

  return res.status(500)
    .json({ error: "internal_error", message: "An unexpected error occurred." });
}

// Register after all routes:
app.use(cyclesErrorHandler);
```

For per-route handling with the `cyclesGuard` middleware pattern, see the [Express Middleware example](https://github.com/runcycles/cycles-client-typescript/tree/main/examples/express-middleware).

## Next.js API route error handling

In Next.js App Router routes, catch errors and return appropriate responses:

```typescript
import { BudgetExceededError, CyclesProtocolError } from "runcycles";

export async function POST(req: Request) {
  try {
    const result = await handleChat(req);
    return new Response(JSON.stringify(result), {
      headers: { "Content-Type": "application/json" },
    });
  } catch (err) {
    if (err instanceof BudgetExceededError) {
      return new Response(
        JSON.stringify({ error: "budget_exceeded", message: "Budget limit reached." }),
        { status: 402, headers: { "Content-Type": "application/json" } },
      );
    }
    if (err instanceof CyclesProtocolError && err.isDebtOutstanding()) {
      return new Response(
        JSON.stringify({ error: "service_unavailable", message: "Service paused." }),
        { status: 503, headers: { "Content-Type": "application/json" } },
      );
    }
    throw err;
  }
}
```

## Graceful degradation with caps

When the budget system returns `ALLOW_WITH_CAPS`, the decision includes caps that constrain execution. Use these to fall back to cheaper models or limit output:

```typescript
import { withCycles, getCyclesContext, isToolAllowed } from "runcycles";

const callLlm = withCycles(
  { estimate: (prompt: string) => estimateCost(prompt), client, actionKind: "llm.completion", actionName: "gpt-4o" },
  async (prompt: string) => {
    const ctx = getCyclesContext();

    // Respect max tokens cap
    let maxTokens = 4096;
    if (ctx?.caps?.maxTokens) {
      maxTokens = Math.min(maxTokens, ctx.caps.maxTokens);
    }

    // Check tool allowlist
    const tools = allTools.filter((t) => {
      if (!ctx?.caps) return true;
      return isToolAllowed(ctx.caps, t.name);
    });

    return openai.chat.completions.create({
      model: "gpt-4o",
      max_tokens: maxTokens,
      messages: [{ role: "user", content: prompt }],
      tools,
    });
  },
);
```

## Retry-after with exponential backoff

When a protocol error includes `retryAfterMs`, use it as the minimum delay before retrying:

```typescript
import { CyclesProtocolError } from "runcycles";

async function withRetry<T>(fn: () => Promise<T>, maxAttempts = 3): Promise<T> {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      if (err instanceof CyclesProtocolError && err.isRetryable() && attempt < maxAttempts) {
        const baseDelay = err.retryAfterMs ?? 1000 * Math.pow(2, attempt - 1);
        await new Promise((resolve) => setTimeout(resolve, baseDelay));
        continue;
      }
      throw err;
    }
  }
  throw new Error("Unreachable");
}
```

## Distinguishing retryable vs non-retryable errors

Use `isRetryable()` to check whether an error warrants a retry:

```typescript
import { CyclesProtocolError } from "runcycles";

try {
  result = await guardedFunc();
} catch (err) {
  if (err instanceof CyclesProtocolError) {
    if (err.isRetryable()) {
      // INTERNAL_ERROR, UNKNOWN, or 5xx — safe to retry with backoff
      return retryLater(err.retryAfterMs);
    }
    if (err.isBudgetExceeded()) {
      // Budget may free up — retry with backoff or degrade
      return fallback();
    }
    // RESERVATION_EXPIRED, IDEMPOTENCY_MISMATCH, etc. — do not retry
    throw err;
  }
  throw err;
}
```

## Next steps

- [Error Handling Patterns](/how-to/error-handling-patterns-in-cycles-client-code) — general error handling patterns across all languages
- [Error Codes and Error Handling](/protocol/error-codes-and-error-handling-in-cycles) — protocol error code reference
- [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) — strategies for handling budget constraints
- [Getting Started with the TypeScript Client](/quickstart/getting-started-with-the-typescript-client) — TypeScript client setup
- [Testing with Cycles](/how-to/testing-with-cycles) — testing patterns for Cycles-governed code
- [SDK Settlement Recovery and Durability](/protocol/sdk-settlement-recovery-and-durability) — durable replay, expiry fallback, and guarantee boundary



# Evaluate Cycles for multi-tenant agents

This page is for engineering and product leads building multi-tenant agents that call models, tools, APIs, or external systems. It is **not** an implementation guide — for that, see [Building a Multi-Tenant AI SaaS with Cycles](/how-to/multi-tenant-saas-with-cycles) and [Choosing the Right Integration Pattern](/how-to/choosing-the-right-integration-pattern).

The goal here is to answer one question: **does Cycles solve a problem you are about to have?**

## When Cycles is a fit

Cycles is built for agent systems where the *next* tool call, model call, or external action might be the one that costs you money, breaks a tenant boundary, or commits an irreversible side effect. You probably want it if any of these describe your stack:

- Agents call **paid APIs** (LLMs, search, browser automation, third-party data) where cost compounds with retries.
- Agents call **tools with side effects** — sending email, writing to databases, triggering deployments, executing shell commands, mutating customer state.
- You serve **multiple tenants** and need each one capped independently. One customer's runaway agent must not eat another customer's budget or quota.
- Workflows **retry, fan out, or run unattended**. The first call is fine; the 12th call is the problem.
- Your existing controls are **observation, not enforcement** — alerts, dashboards, traces. By the time the alert fires, the action already happened.
- You need **per-run** or **per-tool** caps that traditional rate limits cannot express. Rate limits control velocity; they do not control cumulative cost or blast radius.

If three or more of these describe you today, the rest of this page is worth your time.

## When Cycles is not a fit

Skip Cycles if:

- You are running a **single-user script** or a **toy agent** with no production exposure.
- Agents make **no paid calls** and trigger **no irreversible actions**.
- There is **no multi-tenant boundary** and no need for per-tenant isolation.
- You only need **logging or analytics** of what already happened — Cycles is a *gate* before execution, not an after-the-fact ledger.
- You want a **managed cloud SaaS**. Cycles is self-hosted today (Apache 2.0), so it fits teams comfortable running the control plane inside their own infrastructure.

## 15-minute local test

The fastest way to know whether Cycles fits is to run the full stack and watch a denial happen.

1. **Start the stack** with the published Docker images. See [Deploying the Full Cycles Stack](/quickstart/deploying-the-full-cycles-stack). You'll have the runtime server and admin server running locally on two ports, backed by Redis (the events service is included but optional). The dashboard is **not** part of this compose stack — if you want a web UI for step 6, deploy the [Cycles Admin Dashboard](/quickstart/deploying-the-cycles-dashboard) separately from its own repo.
2. **Create a tenant** via the admin server. See [Tenant Creation and Management](/how-to/tenant-creation-and-management-in-cycles).
3. **Create a budget** scoped to that tenant — a small one, e.g., a few cents.
4. **Run one allowed check.** A `decide` call returns `ALLOW` without creating a reservation, or a `reserve` call returns `ALLOW` and records an active reservation.
5. **Run one denied check.** Exhaust the tenant budget with a request larger than the remaining balance. A `decide` call returns `DENY`, or a `reserve` call is rejected before the underlying action executes. (If you're evaluating the [v0.1.26 action-governance preview](/protocol/action-governance-preview-in-cycles), you can also test per-action quotas and allow/deny lists.)
6. **Inspect the results.** Query the balances API to see the reservation, commit, and denial reflected in the tenant's ledger — or, if you deployed the optional dashboard, watch them show up under the tenant's budget view. If you used `decide`, expect a decision result but no active reservation.

You should see three things:

- an **allowed reservation** that reduces available budget
- a **committed reservation** that records actual usage
- a **denied check** that prevents the next action from running

If you can map those three states to your own agent workflow, Cycles is probably worth a deeper integration test. If they don't match what you'd expect for your worst case — runaway agent, tool loop, multi-tenant overspend — you've spent 15 minutes and learned something specific about why.

## What to test in your own stack

Once the local test passes, try Cycles against the actions in your product that actually scare you. The list usually looks something like:

- **LLM call** — the most expensive class of action; reservations should align with token budgets.
- **Email / message send** — irreversible side effect; needs a per-run cap regardless of cost.
- **Browser action** — fan-out and retry storms are common; cap per session.
- **Database write** — usually cheap to issue, expensive to undo; gate by risk, not just cost.
- **Deployment / infra command** — high blast radius; should require an explicit allowlist, not just a budget.
- **Coding-agent shell command** — agents that write code will retry shell commands aggressively; cap per run.

For each, decide:

- What action kind does it map to? (`llm.completion`, `web.search`, `message.email.send`, `code.exec.shell`, etc.)
- What should be enforced today — spend, token, or caller-assigned risk budget? Treat action-count quotas as a v0.1.26 preview until the reference server implements them.
- Which standard scope owns the ledger — tenant, workspace, app, workflow, agent, or toolset? Map a run to a unique workflow value when needed.

That mapping is the design exercise. See [Assigning Risk Points to Agent Tools](/how-to/assigning-risk-points-to-agent-tools) for the framework, and [Choosing the Right Integration Pattern](/how-to/choosing-the-right-integration-pattern) for where to put the gate (SDK in-process, MCP, gateway, framework plugin).

## The architecture in one sentence

Cycles becomes a **runtime budget-authority layer** between agent intent and external execution when every selected consequential path passes through `reserve → execute → commit` (or `release` on failure), using the standard subject scopes. Application authorization still decides whether the tool and arguments are permitted.

That is the core idea. Multi-tenant isolation, per-tier budgets, [action-governance previews](/protocol/action-governance-preview-in-cycles), OTLP metrics, MCP integration, and dashboard workflows all build on that one reserve-before-execute boundary.

## What a good first integration looks like

Do not start by gating every action.

Start with one high-signal boundary:

- one tenant
- one workflow
- one risky action kind
- one small budget, or one quota if you're evaluating the v0.1.26 preview
- one visible denial in the dashboard

Good first candidates are email sends, browser actions, coding-agent shell commands, paid search/API calls, or expensive LLM completions. Once that path works, expand to more tools and scopes.

## Send us your flow

If you want a sanity check before you start, paste the rough shape of your agent's tool-call flow:

```
agent → tool → API → side effect
```

Send it via [Contact Us](/contact) with the subject "agent flow review." We'll mark where `reserve`, `commit`, and `release` belong — or tell you if Cycles is not the right fit. Honest answers, not sales calls.

## Next steps

If Cycles is a fit:

- [Building a Multi-Tenant AI SaaS with Cycles](/how-to/multi-tenant-saas-with-cycles) — the full implementation guide.
- [Deploying the Full Cycles Stack](/quickstart/deploying-the-full-cycles-stack) — the local stack you'll evaluate against.
- [Choosing the Right Integration Pattern](/how-to/choosing-the-right-integration-pattern) — SDK vs MCP vs gateway vs plugin.
- [Add Cycles with Claude or Codex](/how-to/add-cycles-with-claude-or-codex) — fastest path if you're using a coding agent.

If you want to read more before deciding:

- [What is Cycles?](/quickstart/what-is-cycles) — overview with a code sample.
- [Why rate limits are not enough for autonomous systems](/concepts/why-rate-limits-are-not-enough-for-autonomous-systems) — the conceptual case.
- [Runaway agents and the incidents Cycles prevents](/incidents/runaway-agents-tool-loops-and-budget-overruns-the-incidents-cycles-is-designed-to-prevent) — real failure modes.



# Force-Releasing Stuck Reservations as an Operator

Reservations can get stuck. A client crashes without committing. A network partition hangs a request indefinitely. A bug in an agent leaves a reservation open past its intended lifetime. The budget stays reserved, and — because `remaining = allocated - spent - reserved - debt` — other work starts getting denied even though no actual money was spent.

The normal path is to wait for the reservation to expire (`ttl_ms` plus the grace period) and let the server-side sweeper release it. That is usually fine. But during an incident you often cannot wait: customer-facing agents are failing reservation requests *right now* because budget is parked on a hung run that has already been cancelled upstream.

For this case the runtime plane supports **admin-on-behalf-of release** on `POST /v1/reservations/{id}/release` (added in `cycles-server` v0.1.25.8). Operators can force-release any reservation with `X-Admin-API-Key`; no tenant-scoped API key is required. Every such call is tagged in the audit log so the action is never invisible.

::: tip When to reach for this
Use force-release only when you cannot wait for normal expiry and you have confirmed the downstream work is truly dead. Releasing a reservation whose client is still executing will cause the client's commit to land on a freshly-restored balance — double-counting. If there is any doubt, wait for expiry.
:::

## Admin-authenticated request

The force-release call is the standard release endpoint authenticated with the runtime server's admin key:

```bash
curl -X POST "http://localhost:7878/v1/reservations/rsv_abc123/release" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "idempotency_key": "incident-2026-04-17-release-rsv_abc123",
    "reason": "Force-release during incident INC-842; client confirmed dead by oncall"
  }'
```

The runtime server:

1. Validates the `X-Admin-API-Key` against its local `ADMIN_API_KEY` env var (must match the admin server's key — deploy them with the same value).
2. Looks up the reservation owner from the reservation ID and skips tenant-key ownership checks because this is the admin-on-behalf-of path.
3. Runs the `release.lua` script to return `reserved` budget to the pool.
4. Writes an audit entry with `operation = releaseReservation`, `resource_type = reservation`, `resource_id = <reservation_id>`, and `metadata.actor_type = admin_on_behalf_of`.

Response is identical to a normal release:

```json
{
  "status": "RELEASED",
  "released": { "amount": 5000, "unit": "USD_MICROCENTS" },
  "balances": []
}
```

::: warning ADMIN_API_KEY must be the same on both planes
Admin-on-behalf-of authenticates on the runtime plane, so `cycles-server` and `cycles-server-admin` must share the same `ADMIN_API_KEY`. If the runtime plane has a different value, the request returns `401 UNAUTHORIZED`. If the runtime plane has no admin key configured, the request fails closed as a server misconfiguration and the release does not happen.
:::

## Finding the reservation to release

You usually do not know the `reservation_id` off the top of your head. Two common paths to find it:

### From an idempotency key

If the client was built to generate idempotency keys for reservations before calling reserve (which it should be — that is the canonical recovery pattern), you can look up the reservation from the key the application last logged:

```bash
curl -G "http://localhost:7878/v1/reservations" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  --data-urlencode "tenant=acme-corp" \
  --data-urlencode "idempotency_key=run-ac35f-step-4" | jq .
```

Idempotency keys are unique per `(tenant, endpoint, key)`, so this returns at most one match.

### From a filtered listing

If the client did not log the key, find it by subject and status:

```bash
curl -G "http://localhost:7878/v1/reservations" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  --data-urlencode "tenant=acme-corp" \
  --data-urlencode "status=ACTIVE" \
  --data-urlencode "app=support-bot" \
  --data-urlencode "sort_by=expires_at_ms" \
  --data-urlencode "sort_dir=asc" \
  --data-urlencode "limit=20" | jq .
```

Oldest-expiring first is the most operationally useful view — hung reservations usually have `expires_at_ms` well in the past. See [Reservation Recovery and Listing](/protocol/reservation-recovery-and-listing-in-cycles#sorting-v0-1-25-12) for the full sort parameter catalog.

## Audit trail

Every force-release is captured in the audit log. Query it from the admin plane:

```bash
curl -G "http://localhost:7979/v1/admin/audit/logs" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  --data-urlencode "operation=releaseReservation" \
  --data-urlencode "resource_type=reservation" \
  --data-urlencode "resource_id=rsv_abc123" | jq .
```

Fields to expect on the audit entry:

| Field | Meaning |
|-------|---------|
| `operation` | `releaseReservation` |
| `resource_type` | `reservation` |
| `resource_id` | The reservation ID |
| `metadata.actor_type` | `admin_on_behalf_of` (distinguishes from tenant-initiated releases) |
| `tenant_id` | Tenant the reservation belonged to |
| `metadata.reason` | Free-form text from the request body, sanitized for CR/LF if provided |
| `request_id` | The HTTP request id for the release call |
| `trace_id` | W3C-compatible trace ID for joining the audit entry to response and application logs |

The audit entry is retained under the authenticated-audit retention policy (default 400 days in v0.1.25.20+). This comfortably covers SOC2 audit windows.

### Joining the release audit entry with `trace_id`

Every response from the runtime server — including the force-release response — carries an `X-Cycles-Trace-Id` header, and error bodies carry a `trace_id` field (v0.1.25.14+). Use the response header to retrieve the matching audit entry:

```bash
TID=<32-hex from X-Cycles-Trace-Id response header>

# The audit entry for this specific release
curl -s "http://localhost:7979/v1/admin/audit/logs?trace_id=$TID" \
  -H "X-Admin-API-Key: $ADMIN_KEY"
```

The current reference runtime does not emit `reservation.released`, so an ordinary or force-release operation does not produce a release webhook to confirm. Join the audit row to your incident and application logs using the same trace ID. The admin `trace_id` audit filter requires `cycles-server-admin` v0.1.25.31+. See [Correlation and Tracing](/protocol/correlation-and-tracing-in-cycles).

## Recovery checklist

A pragmatic runbook for hung-reservation incidents:

1. **Confirm the client is dead.** Check logs, look for the idempotency key, grep for the reservation ID in traces. If the client is still running, do not force-release — either wait for TTL or release from the client itself.
2. **Find the reservation.** Use idempotency key lookup first, filtered listing second.
3. **Inspect it.** `GET /v1/reservations/{id}` returns the full state — status, subject, reserved amount, TTL, and finalized time (null if still ACTIVE).
4. **Decide: release vs wait.** If `expires_at_ms` is within minutes, waiting is safer. If it is hours away and budget is being denied right now, force-release.
5. **Force-release with a reason.** Include the incident ID and the reason in both `idempotency_key` and `reason` — the audit entry is the record of why.
6. **Verify.** Re-fetch the reservation; it should now be `RELEASED` with a `finalized_at_ms`.
7. **Post-incident.** Document the hung reservation in the incident report. If the pattern repeats, look at shortening client TTLs or adding heartbeat extension.

::: tip Dashboard equivalent
The Reservations page in the [Cycles Admin Dashboard](/quickstart/deploying-the-cycles-dashboard) exposes this exact flow — filter to `status=ACTIVE`, sort by `expires_at_ms` ascending, click a row, and use the **Force Release** button. The dashboard sends the admin-authenticated call through its nginx reverse proxy to the runtime plane.
:::

## Failure modes

- **401 UNAUTHORIZED** — the admin key is wrong.
- **403 FORBIDDEN** — you used tenant-key auth and that tenant does not own the reservation. Under `X-Admin-API-Key`, admin-on-behalf-of skips tenant ownership checks on this allowlisted endpoint.
- **500 INTERNAL_ERROR** — the runtime server does not have `ADMIN_API_KEY` configured for the admin-on-behalf-of path.
- **404 NOT_FOUND** — the reservation never existed.
- **409 RESERVATION_FINALIZED** — the reservation is already `COMMITTED` or `RELEASED`.
- **410 RESERVATION_EXPIRED** — the reservation already expired before you got to it; the budget is already back.

## Next steps

- [Reservation Recovery and Listing](/protocol/reservation-recovery-and-listing-in-cycles) — listing and sort parameters
- [Production Operations Guide](/how-to/production-operations-guide) — broader runbook patterns
- [Admin API reference](/admin-api/) — audit log query endpoints
- [Security Hardening](/how-to/security-hardening) — protecting the admin key across planes



# Handling Streaming Responses with Cycles

Streaming LLM responses require special handling because the actual cost is only known after the stream completes. This guide shows the reserve → stream → commit pattern.

## The challenge

With non-streaming calls, the `@cycles` decorator handles the full lifecycle automatically. Streaming needs different handling because:

1. The reservation must stay alive for the duration of the stream
2. Token counts accumulate incrementally
3. If the stream fails mid-way, you should release the reservation

## The pattern

### Python

Use `client.stream_reservation()`, a context manager that reserves budget on enter, auto-commits the actual cost on successful exit, and auto-releases on exception:

```python
from openai import OpenAI
from runcycles import Action, Amount, CyclesClient, CyclesConfig, Unit

client = CyclesClient(CyclesConfig.from_env())
openai_client = OpenAI()

PRICE_PER_INPUT_TOKEN = 250
PRICE_PER_OUTPUT_TOKEN = 1_000

def stream_with_budget(prompt: str, max_tokens: int = 1024) -> str:
    estimated_cost = max_tokens * PRICE_PER_OUTPUT_TOKEN  # worst case

    with client.stream_reservation(
        action=Action(kind="llm.completion", name="gpt-4o"),
        estimate=Amount(unit=Unit.USD_MICROCENTS, amount=estimated_cost),
        cost_fn=lambda u: u.tokens_input * PRICE_PER_INPUT_TOKEN
                          + u.tokens_output * PRICE_PER_OUTPUT_TOKEN,
    ) as reservation:
        # Respect budget caps
        if reservation.caps and reservation.caps.max_tokens:
            max_tokens = min(max_tokens, reservation.caps.max_tokens)

        stream = openai_client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": prompt}],
            max_tokens=max_tokens,
            stream=True,
            stream_options={"include_usage": True},
        )

        chunks = []
        for chunk in stream:
            if chunk.choices and chunk.choices[0].delta.content:
                chunks.append(chunk.choices[0].delta.content)
            if chunk.usage:  # the final chunk includes usage stats
                reservation.usage.tokens_input = chunk.usage.prompt_tokens
                reservation.usage.tokens_output = chunk.usage.completion_tokens

    # Auto-committed on exit with the actual cost computed by cost_fn
    return "".join(chunks)
```

The context manager handles the full lifecycle:

- **On enter** — creates the reservation (default `ttl_ms=120_000`). A DENY or protocol error raises `CyclesProtocolError` (or a subclass such as `BudgetExceededError`), so the stream never starts without budget.
- **During the stream** — a background heartbeat schedules from server-authoritative `remaining_ttl_ms` when present and uses a best-effort fallback for older servers (background thread; asyncio task in the async variant). Update `reservation.usage` (`tokens_input`, `tokens_output`, or `set_actual_cost()`) as chunks arrive.
- **On exit** — commits the actual cost: an explicit `set_actual_cost()` value wins, then `cost_fn(usage)`, then the estimate as fallback. If the body raised, the reservation is released instead; see the partial-usage warning below.

The subject defaults to the `CyclesConfig` subject fields; pass `subject=Subject(...)` to override. With `AsyncCyclesClient`, the same call returns an async context manager: `async with client.stream_reservation(...) as reservation:`.

### Python: manual control

Under the hood, `stream_reservation` drives the raw reserve → stream → commit/release calls. Use them directly only when you need control the context manager doesn't offer:

```python
import uuid
from openai import OpenAI
from runcycles import (
    CyclesClient, CyclesConfig, ReservationCreateRequest,
    CommitRequest, ReleaseRequest, Subject, Action, Amount,
    Unit, CyclesMetrics,
)

client = CyclesClient(CyclesConfig.from_env())
openai_client = OpenAI()

def stream_with_budget(prompt: str, max_tokens: int = 1024) -> str:
    key = str(uuid.uuid4())

    # 1. Reserve worst-case budget
    res = client.create_reservation(ReservationCreateRequest(
        idempotency_key=key,
        subject=Subject(tenant="acme", agent="streaming-agent"),
        action=Action(kind="llm.completion", name="gpt-4o"),
        estimate=Amount(unit=Unit.USD_MICROCENTS,
                        amount=max_tokens * 1_000),  # worst case
        ttl_ms=120_000,  # longer TTL for streaming
    ))

    if not res.is_success:
        raise RuntimeError(f"Reservation failed: {res.error_message}")

    # Defensive: a conformant server returns 409 on live budget denial, but
    # dry-run responses (and lenient servers) return 200 with decision=DENY
    # and no reservation_id - check before using it
    if res.get_body_attribute("decision") == "DENY":
        raise RuntimeError(
            f"Reservation denied: {res.get_body_attribute('reason_code')}"
        )

    reservation_id = res.get_body_attribute("reservation_id")

    # 2. Stream, with release on failure
    chunks = []
    try:
        stream = openai_client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": prompt}],
            max_tokens=max_tokens,
            stream=True,
            stream_options={"include_usage": True},
        )

        input_tokens = 0
        output_tokens = 0

        for chunk in stream:
            if chunk.choices and chunk.choices[0].delta.content:
                chunks.append(chunk.choices[0].delta.content)
            if chunk.usage:
                input_tokens = chunk.usage.prompt_tokens
                output_tokens = chunk.usage.completion_tokens

    except Exception:
        # Release budget on failure
        client.release_reservation(
            reservation_id,
            ReleaseRequest(idempotency_key=f"release-{key}"),
        )
        raise

    # 3. Commit actual cost
    actual_cost = input_tokens * 250 + output_tokens * 1_000
    client.commit_reservation(reservation_id, CommitRequest(
        idempotency_key=f"commit-{key}",
        actual=Amount(unit=Unit.USD_MICROCENTS, amount=actual_cost),
        metrics=CyclesMetrics(
            tokens_input=input_tokens,
            tokens_output=output_tokens,
            custom={"streamed": True},
        ),
    ))

    return "".join(chunks)
```

### TypeScript

The TypeScript client provides `reserveForStream`, which handles reservation creation and automatic heartbeat (TTL extension) in one call:

```typescript
import OpenAI from "openai";
import {
  CyclesClient,
  CyclesConfig,
  reserveForStream,
  BudgetExceededError,
} from "runcycles";

const cyclesClient = new CyclesClient(CyclesConfig.fromEnv());
const openai = new OpenAI();

async function streamWithBudget(
  prompt: string,
  maxTokens = 1024,
): Promise<string> {
  // 1. Reserve budget (starts automatic heartbeat)
  const handle = await reserveForStream({
    client: cyclesClient,
    estimate: maxTokens * 1000, // worst-case output cost
    unit: "USD_MICROCENTS",
    actionKind: "llm.completion",
    actionName: "gpt-4o",
  });

  try {
    // Respect budget caps
    let effectiveMaxTokens = maxTokens;
    if (handle.caps?.maxTokens) {
      effectiveMaxTokens = Math.min(maxTokens, handle.caps.maxTokens);
    }

    // 2. Stream the response
    const stream = await openai.chat.completions.create({
      model: "gpt-4o",
      messages: [{ role: "user", content: prompt }],
      max_tokens: effectiveMaxTokens,
      stream: true,
      stream_options: { include_usage: true },
    });

    const chunks: string[] = [];
    let inputTokens = 0;
    let outputTokens = 0;

    for await (const chunk of stream) {
      const content = chunk.choices[0]?.delta?.content;
      if (content) chunks.push(content);
      if (chunk.usage) {
        inputTokens = chunk.usage.prompt_tokens ?? 0;
        outputTokens = chunk.usage.completion_tokens ?? 0;
      }
    }

    // 3. Commit actual cost (stops heartbeat automatically)
    const actualCost = Math.ceil(inputTokens * 250 + outputTokens * 1000);
    await handle.commit(actualCost, {
      tokensInput: inputTokens,
      tokensOutput: outputTokens,
    });

    return chunks.join("");
  } catch (err) {
    // Release budget on failure (stops heartbeat automatically)
    await handle.release("stream_error");
    throw err;
  }
}
```

`reserveForStream` handles TTL extension automatically via a background heartbeat, so you don't need to call `extend` manually. The heartbeat stops when you call `commit` or `release`.

## TTL considerations

Streaming responses can take significantly longer than non-streaming calls. Set `ttl_ms` high enough to cover the full stream duration:

| Response size | Suggested TTL |
|--------------|---------------|
| Short (< 500 tokens) | 30,000 ms |
| Medium (500–2000 tokens) | 60,000 ms |
| Long (> 2000 tokens) | 120,000 ms |

`stream_reservation` schedules extensions automatically from the server's remaining-lifetime field when available, the same way the decorator does. Manual extension is only needed when you drive raw `create_reservation` calls yourself:

```python
from runcycles import ReservationExtendRequest

# Extend by another 60 seconds
client.extend_reservation(
    reservation_id,
    ReservationExtendRequest(
        idempotency_key=f"extend-{key}",
        extend_by_ms=60_000,
    ),
)
```

Raw clients must implement the [normative heartbeat schedule and same-key recovery rules](/protocol/reservation-ttl-grace-period-and-extend-in-cycles), not a blind half-TTL interval. Generate a fresh key for a new logical extension and reuse that key while recovering an ambiguous result.

## Release on failure

Release the reservation if streaming fails before any billable usage occurs. This frees held budget immediately rather than waiting for TTL expiry. In a manual lifecycle where the provider never began work:

```python
try:
    # stream...
except Exception:
    client.release_reservation(
        reservation_id,
        ReleaseRequest(idempotency_key=f"release-{key}"),
    )
    raise
```

::: warning Partial provider usage
If the provider billed tokens before the stream failed, releasing would return budget for real spend. Capture the best-known actual amount and settle it instead. Python `stream_reservation` releases whenever an exception escapes its body, so catch the provider error inside the context, set the actual cost, let the context exit cleanly to commit, then re-raise the saved error afterward. The SDK cannot recover an amount it never receives; applications needing crash/failure convergence must durably checkpoint provider receipts or usage before acknowledging the stream.
:::

## Respecting caps

With `stream_reservation`, caps are available as `reservation.caps` immediately after entering the context (see the primary example above). In the manual pattern, check the raw response:

```python
caps = res.get_body_attribute("caps")
if caps and caps.get("max_tokens"):
    max_tokens = min(max_tokens, caps["max_tokens"])
```

## Estimating accurately

The estimate determines how much budget is held. Over-estimating wastes budget capacity; under-estimating risks commit-time overage errors.

For streaming, a good estimate is `max_tokens × output_price`, since output tokens dominate cost and `max_tokens` is the upper bound.

## Key points

- **Use `stream_reservation`**, not the decorator, for streaming in Python — it handles reserve, heartbeat, commit, and release for you.
- **Set a longer TTL** to cover the full stream duration.
- **Release only unused reservations**; settle best-known actual usage when the provider already billed part of a failed stream.
- **Commit the actual cost** after the stream completes using usage data from the final chunk.
- **The estimate holds budget** — the difference between estimate and actual is freed at commit time.
- **Known-actual settlement is durable** in current lifecycle helpers; see [SDK Settlement Recovery and Durability](/protocol/sdk-settlement-recovery-and-durability).

## Full example

See [`examples/streaming_usage.py`](https://github.com/runcycles/cycles-client-python/blob/main/examples/streaming_usage.py) for a complete, runnable script.

## Java / Spring Boot

The Spring Boot starter's `@Cycles` annotation does not support streaming responses. For streaming in Java, use the programmatic `CyclesClient` directly with the reserve → stream → commit pattern shown in the Python manual-control section above. See the [Spring Boot starter overview](/quickstart/getting-started-with-the-cycles-spring-boot-starter) for the broader integration model.

## Next steps

- [Error Handling Patterns in Python](/how-to/error-handling-patterns-in-python) — handling failures during streaming
- [Reservation TTL, Grace Period, and Extend](/protocol/reservation-ttl-grace-period-and-extend-in-cycles) — configuring timeouts for long-running streams
- [Cost Estimation Cheat Sheet](/how-to/cost-estimation-cheat-sheet) — estimating token costs for budget reservations
- [Integrating Cycles with Express](/how-to/integrating-cycles-with-express) — Express.js streaming with `reserveForStream`
- [Integrating Cycles with FastAPI](/how-to/integrating-cycles-with-fastapi) — FastAPI streaming with the programmatic client



# How to Add Budget Control to a LangChain Agent

This walkthrough gates a LangChain 1.x agent at all three execution boundaries:
model calls, tool side effects, and repeated model turns.

## 1. Install and configure

```bash
pip install "langchain-runcycles>=0.4.0" langchain-openai

export CYCLES_BASE_URL="http://localhost:7878"
export CYCLES_API_KEY="your-api-key"
export CYCLES_TENANT="acme"
export OPENAI_API_KEY="sk-..."
```

Create the tenant budget and API key before running the agent. See
[Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack) for a local
server setup.

## 2. Build the gated agent

```python
from langchain.agents import create_agent
from langchain.tools import tool
from langchain_runcycles import CyclesFanOutGate, CyclesModelGate, CyclesToolGate
from langchain_runcycles.extractors import openai_cost
from runcycles import Action, Amount, CyclesClient, CyclesConfig, Subject, Unit

client = CyclesClient(CyclesConfig.from_env())
subject = Subject(tenant="acme", workflow="collections", agent="notifier")

@tool
def send_notice(account_id: str, body: str) -> str:
    """Send one collections notice."""
    return f"sent:{account_id}"

agent = create_agent(
    model="gpt-4o",
    tools=[send_notice],
    middleware=[
        CyclesFanOutGate(
            max_turns=12,
            client=client,
            subject=subject,
            action=Action(kind="model.turn", name="collections"),
        ),
        CyclesModelGate(
            client,
            subject=subject,
            action=Action(kind="llm.completion", name="gpt-4o"),
            mode="decide+reserve",
            estimate=Amount(unit=Unit.USD_MICROCENTS, amount=2_000_000),
            cost_fn=openai_cost(
                prompt_per_million_usd=2.50,
                cached_prompt_per_million_usd=1.25,
                completion_per_million_usd=10.00,
            ),
        ),
        CyclesToolGate(
            client,
            subject=subject,
            action={"send_notice": Action(kind="tool.call", name="send_notice")},
            mode="decide+reserve",
            estimate=Amount(unit=Unit.RISK_POINTS, amount=1),
            idempotency_namespace="collections-run-123",
            settlement_error_policy="log",
        ),
    ],
)
```

The model reservation limits cost. The tool reservation uses `RISK_POINTS` so
operators can independently cap side-effect exposure. `settlement_error_policy`
is `"log"` for the notice because an agent retry must not send it twice; durable
settlement recovery still remains queued.

## 3. Invoke and handle denials

```python
result = agent.invoke({
    "messages": [{"role": "user", "content": "Send the approved notice."}],
    "run_id": "collections-run-123",
})
```

When a model reservation is denied, the middleware returns a terminal
`ModelResponse`. When the tool is denied, the model receives a correlated
`ToolMessage`, and `send_notice` never runs. `ALLOW_WITH_CAPS` is also an allowed
decision; apply relevant caps in your host or model configuration.

At the raw API boundary, insufficient reserve returns HTTP 409
`BUDGET_EXCEEDED`. The middleware converts it to the LangChain denial result;
do not write low-level code that treats only 2xx responses as possible denials.

## 4. Understand the failure behavior

Reserve mode is not a three-call teaching snippet. While the handler runs, the
Python SDK heartbeats the lease. After success it journals the exact commit
before the first network attempt. Transient failure survives process restart,
and an expired reservation transitions to an idempotent `/v1/events` recovery.

Only handler failure before an actual is recorded uses release. A commit error
after the action ran never releases known spend.

For completed model streams, LangChain supplies final aggregated normalized
usage and the same settlement path applies. A cancelled partial stream has no
final usage object; reconcile any provider charge from provider telemetry.

## 5. Validate before production

- Use a run-scoped `idempotency_namespace`; do not hard-code one across all runs.
- Keep tool side effects idempotent even when the Cycles reservation key is stable.
- Verify the price rates for the exact model and cache tier you deploy.
- Exercise deny, timeout, process-restart, expired-reservation, and stream-cancel paths.
- Monitor the SDK commit journal and settlement warnings.

## Next steps

- [Full LangChain integration guide](/how-to/integrating-cycles-with-langchain)
- [Caps and the three-way decision model](/protocol/caps-and-the-three-way-decision-model-in-cycles)
- [SDK settlement recovery and durability](/protocol/sdk-settlement-recovery-and-durability)



# How to Estimate Exposure Before Execution: Practical Reservation Strategies for Cycles

One of the first practical questions teams ask when adopting Cycles is:

**How do we know how much to reserve before the work actually runs?**

That is the right question.

Cycles requires a system to reserve bounded exposure before execution, then commit actual usage afterward. But many real workloads do not know their exact final cost in advance.

A model call may return sooner than expected.  
A tool path may branch differently.  
A workflow may exit early.  
A retry may or may not happen.  
An agent may decide not to invoke a tool after all.

This is normal.

Cycles does not require perfect prediction.  
It requires a reservation strategy that is **good enough to bound execution before work starts**, while still allowing accurate reconciliation afterward.

This article explains practical ways to estimate exposure before execution, how to think about over-reserving versus under-reserving, and how teams can improve reservation quality over time.

## The goal is not perfect prediction

A common mistake is to think reservation only works if the system can predict exact usage in advance.

That is not how real systems behave.

Reservation is not a prophecy.  
It is a **bounded allowance**.

The system is asking:

::: info
How much room should this action be allowed to consume before it proceeds?
:::

That means a useful estimate should be:

- directionally reasonable
- safe enough for the action type
- tied to likely execution shape
- reconcilable against actual usage later

Perfect precision is not required.

## Why estimation matters

The reservation amount affects both safety and usability.

If you reserve too little:

- the system may deny actions that should have been allowed
- work may run out of budget too early
- expensive steps may not have enough room to complete safely
- policy may become brittle

If you reserve too much:

- too much budget is tied up temporarily
- other work may be denied unnecessarily
- tenant or run pressure may appear higher than it really is
- capacity may look artificially constrained

This is why reservation quality matters, even if perfection is impossible.

## A simple mental model

Think of reservation as an **execution envelope**.

The estimate does not need to equal the exact final usage.

It needs to define a bounded space in which the action is allowed to operate.

Then, once execution finishes:

- actual usage is committed
- unused remainder is released

That means reservation quality should be judged by whether it creates a usable and safe envelope, not by whether it predicts the future exactly.

## The main reservation strategies

There is no single correct estimation method.

Most teams start with one or more of these strategies:

- fixed reservation
- class-based reservation
- heuristic reservation
- historical percentile reservation
- stepwise reservation
- conservative plus degradation reservation

Each has tradeoffs.

## 1. Fixed reservation

This is the simplest strategy.

Every action of a given type reserves the same amount.

Examples:

- every chat model call reserves 100 units
- every web search tool call reserves 25 units
- every workflow step reserves 50 units

### Why fixed reservation is useful

It is easy to explain and easy to implement.

It is often the best first strategy when:

- the team is just starting
- the action shape is fairly stable
- the first goal is getting Cycles into production
- shadow mode is being used to calibrate later

### Limitations

Fixed reservation can be too blunt when action cost varies widely.

It may lead to:

- chronic over-reservation for cheap actions
- under-reservation for expensive ones
- poor fit for broad workflow classes

Still, it is often a very good first rollout.

## 2. Class-based reservation

This strategy uses different fixed reservations for different classes of actions.

Examples:

- small-model call = 40 units
- large-model call = 120 units
- read-only tool = 20 units
- external write-capable tool = 80 units
- short workflow step = 30 units
- high-risk workflow step = 150 units

### Why class-based reservation is useful

It is more expressive than one flat number, while still staying operationally simple.

This is often the best next step after fixed reservation.

It works well when the team can identify clear categories such as:

- model size
- tool type
- side-effect level
- workflow tier
- risk class

### Limitations

Class definitions can drift over time, and some actions still vary a lot within a class.

But for many systems, class-based reservation offers a strong balance between simplicity and control.

## 3. Heuristic reservation

This strategy estimates exposure based on properties known before execution.

Examples:

- expected token count
- prompt length
- number of tool candidates
- workflow type
- whether retrieval is enabled
- whether external side effects are possible
- current phase of the run
- whether the model is large or small

### Why heuristic reservation is useful

It allows reservation to respond to real input shape.

For example:

- larger prompts may reserve more
- tool-enabled calls may reserve more than model-only calls
- workflows with write capability may reserve more than read-only workflows

### Limitations

Heuristics can become overly complicated if the team tries to model too many variables too early.

A good heuristic should improve safety and fit without turning into a fragile prediction engine.

## 4. Historical percentile reservation

This strategy uses observed past usage to set reservation levels.

For example:

- reserve at the 90th percentile of prior actual usage for this action class
- reserve at the 95th percentile for high-side-effect actions
- reserve based on rolling usage history for a workflow type

### Why historical percentile reservation is useful

It is grounded in real system behavior.

This often works well once the team has enough observed usage from:

- shadow mode
- production logs
- stable workflow patterns
- repeated model/tool usage classes

### Limitations

Historical strategies can be misleading when:

- workflows change rapidly
- new action types have little history
- usage distributions are unstable
- tail-risk actions matter more than the average

Still, percentile-based reservation is often one of the best ways to improve estimate quality over time.

## 5. Stepwise reservation

This strategy reserves separately at each step rather than trying to reserve one large amount for an entire run upfront.

Examples:

- reserve for the next model call
- commit actual usage
- reserve again for the next tool step
- repeat as the workflow evolves

### Why stepwise reservation is useful

It matches the reality that many autonomous workflows unfold incrementally.

The system often knows much more about the next step than the entire future path.

This can reduce unnecessary over-reservation.

It is especially useful for:

- agent loops
- tool-calling systems
- branching workflows
- uncertain execution paths

### Limitations

Stepwise reservation introduces more lifecycle events and may require tighter integration with the runtime.

But in many autonomous systems, it is the most realistic strategy.

## 6. Conservative plus degradation reservation

This strategy intentionally reserves conservatively, then uses degradation when reservation fails.

Examples:

- reserve enough for the high-quality path
- if reservation fails, retry with a smaller model estimate
- if reservation still fails, disable optional tools
- if reservation still fails, deny or defer

### Why this is powerful

It allows the system to combine estimation with policy.

Instead of trying to predict one perfect number, the platform can ask:

- is the premium path affordable?
- if not, is the cheaper path affordable?
- if not, should optional capability be removed?
- if not, should execution stop?

This is often a more robust operational model than insisting on one “correct” estimate.

## Choosing the right strategy

A simple rule works well:

- use **fixed** when you need the fastest first rollout
- use **class-based** when action categories are clear
- use **heuristics** when pre-execution signals are meaningful
- use **historical percentiles** when behavior is stable and measured
- use **stepwise** when workflows evolve dynamically
- use **conservative plus degradation** when multiple execution paths exist

In practice, many mature systems combine several of these.

## A practical rollout path

Most teams should not begin with sophisticated estimation.

A strong rollout path is:

### Phase 1: Fixed or class-based reservation

Get reserve → commit / release working first.

### Phase 2: Shadow measurement

Compare estimates with actual usage.

Look for:

- chronic over-reservation
- chronic under-reservation
- high-variance action classes
- workflows with unstable cost shape

### Phase 3: Add simple heuristics or percentiles

Improve the biggest mismatches first.

Do not optimize everything at once.

### Phase 4: Add degradation-aware reservation

Introduce cheaper fallback paths when premium paths cannot reserve enough budget.

This is usually the point where estimation becomes part of a broader control strategy instead of a standalone numeric exercise.

## How to think about over-reserving vs under-reserving

This is one of the most important design decisions.

### Over-reserving

Over-reserving is safer in the sense that work is less likely to run out of budget unexpectedly.

But it can also create:

- false pressure on shared scopes
- unnecessary denials
- poor concurrency utilization
- operator confusion about apparent scarcity

### Under-reserving

Under-reserving makes budget look more available than it really is.

But it can also lead to:

- actions getting partway through without enough room
- repeated reservation failures mid-run
- weaker protection against expensive paths
- policy that appears permissive but is actually fragile

### The right balance

A good default is:

- slightly conservative for high-risk or high-side-effect actions
- tighter and more efficient for stable low-risk actions

Not every action class should be treated the same.

## Estimation should reflect action shape, not only cost

Another common mistake is to think only in terms of average spend.

A better question is:

::: info
What kind of action is this, and how uncertain is its execution path?
:::

For example:

- a deterministic read-only lookup may need a narrow envelope
- a model call with optional retrieval may need a broader one
- a tool-enabled planning step may need a much broader one
- a side-effecting action may justify conservative reservation even if average cost is low

This is why action shape matters just as much as average usage.

## Good first estimation rules

If a team is starting from scratch, these are often strong defaults:

- use fixed reservation for model calls
- use class-based reservation for tools
- reserve more for write-capable or irreversible actions
- reserve by workflow tier for high-level runs
- use stepwise reservation for multi-step agents
- improve with shadow-mode measurement before adding complexity

This gets you operational value quickly without over-engineering.

## What to measure while improving estimates

A team refining reservation quality should track:

- estimate vs actual ratio
- denial frequency by action class
- unused remainder by action class
- actions that regularly exceed estimate
- workflows with high variance
- pressure at tenant and workflow scopes, including workflow ledgers keyed per run
- degradation frequency when premium reservations fail

These signals tell you where the estimate strategy is helping and where it needs refinement.

## Common mistakes

### Mistake 1: Waiting for perfect estimates before rollout

This delays adoption unnecessarily.

Start with a reasonable envelope and improve from real usage.

### Mistake 2: Using one global reservation number for everything

This is simple, but often too blunt once workflows vary meaningfully.

### Mistake 3: Making estimation logic too complex too early

If the estimation model is hard to explain, it will be hard to trust and operate.

### Mistake 4: Ignoring the difference between risky and merely expensive actions

Side-effecting actions may justify more conservative reservation even when average cost is not high.

### Mistake 5: Failing to compare estimates with actuals

Without feedback, reservation quality does not improve.

## A useful principle

A reservation strategy is good when it helps the system:

- bound execution before work starts
- avoid unnecessary denial
- reconcile cleanly afterward
- remain understandable to operators
- improve over time from observed behavior

That is the standard.

Not prediction perfection.

## Summary

Cycles does not require exact foresight.

It requires a practical way to reserve bounded room for execution before work begins, then reconcile actual usage afterward.

Teams can do that with strategies such as:

- fixed reservation
- class-based reservation
- heuristic estimation
- historical percentile tuning
- stepwise reservation
- conservative reservation plus degradation

The best first strategy is usually the simplest one that creates a safe and understandable execution envelope.

From there, real usage data can make estimates better over time.

That is how reservation becomes operationally useful instead of theoretically perfect.

## Next steps

To explore the Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)



# How to Model Tenant, Workflow, and Run Budgets in Cycles

Once a team understands reserve → commit, the next question is usually:

**What should we actually budget?**

That is where policy design starts.

In Cycles, budgets are not only about total spend. They are about deciding **which scopes matter**, **where limits should be enforced**, and **how autonomous work should be bounded across different levels of execution**.

A useful Cycles deployment usually does not rely on one global budget alone.

Instead, it combines multiple scopes such as:

- tenant
- workspace
- app
- workflow
- agent
- toolset

This article focuses on three useful budgeting patterns:

- **tenant budgets**
- **workflow budgets**
- **run budgets**

Tenant and workflow are native scopes. “Run budget” is an application pattern: use a unique run ID as the `workflow` subject value when each execution needs its own ledger. All protected calls in the run must use that same value.

## Why multiple scopes matter

A single flat budget is simple, but often too blunt.

For example, suppose a platform gives one tenant a daily allowance of 10,000 units.

That may protect the platform overall, but it still leaves important questions unanswered:

- can one workflow consume the entire tenant budget?
- can one runaway execution drain everything?
- should a single run be allowed to recurse indefinitely as long as the tenant still has budget?
- should expensive workflows be bounded differently from cheap ones?

This is why Cycles supports hierarchical budgeting.

Different scopes protect against different failure modes.

## A simple mental model

Think of the scopes like this:

- **Tenant budget** protects the customer or account boundary
- **Workflow budget** protects the logical process boundary
- **Run budget pattern** uses a workflow ledger to protect a single execution boundary

Each one solves a different problem.

### Tenant budget

This answers:

::: info
How much total exposure is this customer allowed to create?
:::

It is your platform-level financial and isolation control.

### Workflow budget

This answers:

::: info
How much exposure is this type of process allowed to consume?
:::

It is your product and feature-level control.

### Run budget

This answers:

::: info
How much submitted exposure can this individual execution consume before the host must stop or degrade protected work?
:::

It is your execution safety control.

All three can matter at the same time.

## Tenant budgets

Tenant budgets are usually the first and most obvious scope.

They are especially important for:

- multi-tenant SaaS systems
- customer-isolated AI platforms
- internal business units
- account-level usage governance

A tenant budget bounds the instrumented exposure submitted against that tenant ledger. It does not cover paths that bypass the integration.

### What tenant budgets are good for

Tenant budgets are good at enforcing:

- daily or monthly spend ceilings
- customer usage isolation
- paid plan boundaries
- hard protection against over-consumption

A tenant budget is usually the scope product and finance teams care about first.

### What tenant budgets do not solve by themselves

Tenant budgets alone do not prevent:

- one runaway workflow consuming the full tenant allowance
- a single bad run doing too much damage before the tenant budget is exhausted
- uneven usage between expensive and cheap workflows
- one feature starving another inside the same tenant boundary

That is why tenant budgets should usually be combined with more local controls.

### Example tenant policy

A useful first tenant policy might be:

- each tenant has a daily budget
- all model calls and tool invocations reserve against it
- when exhausted, high-cost actions stop or downgrade

This creates a hard financial boundary.

## Workflow budgets

Workflow budgets are often the most important scope once autonomous behavior becomes real.

A workflow budget answers:

::: info
How much exposure should this type of process be allowed to consume?
:::

For example, not all workflows are equal.

A support triage workflow might need a larger envelope than:

- a simple summarization task
- a classification step
- a low-cost enrichment action

### Why workflow budgets matter

Workflow budgets prevent one class of work from becoming disproportionately expensive.

They are useful for:

- differentiating premium vs standard workflows
- bounding complex agentic behaviors
- controlling expensive feature paths
- keeping budget policy close to product intent

Without workflow budgets, all usage competes at the tenant level, which is often too coarse.

### Example workflow types

A platform might define workflows like:

- `workflow:support-triage`
- `workflow:refund-assistant`
- `workflow:report-generator`
- `workflow:research-agent`

Each can have a different budget profile.

That allows the platform to say:

- support triage may use more budget than summarization
- research may allow broad search but limited side effects
- refund workflows may permit CRM reads but tightly restrict writes

### Workflow budgets are where product policy becomes operational

This is usually the scope where business meaning becomes budget logic.

It is where teams begin translating product intent into execution boundaries.

## Run budgets

Run budgets are an execution-specific mapping onto a standard scope.

In the Cycles protocol, `run` is not a built-in subject field. Model a run-level budget by setting a unique value such as `workflow="run-12345"`, which derives the scope `workflow:run-12345`. Keep the business workflow name and other attribution in application telemetry or metadata when you use the workflow field for the execution ID.

Do not use the `dimensions` field for run budgets. Scopes are derived only from the six standard subject fields (tenant, workspace, app, workflow, agent, toolset). The `dimensions` map never derives scopes, and servers MAY ignore `dimensions` entirely for budgeting decisions — a run identifier placed there would not be enforced.

A run budget answers:

::: info
How much submitted exposure can this single execution consume before the host must stop or degrade protected work?
:::

This scope is especially important for:

- long-running agents
- recursive tool use
- background jobs
- multi-step workflows
- autonomous loops

### Why run budgets matter

When every costly step crosses a mandatory boundary, a per-run workflow ledger is a strong defense against runaway execution.

Even if the tenant has plenty of remaining budget, one individual run may still need a hard ceiling.

That protects against:

- infinite loops
- excessive retries
- bad planning behavior
- recursive tool chains
- accidental fan-out

### Example run policy

A workflow run might be allowed:

- up to 500 units total
- application-selected downgrade behavior when a reservation is denied or configured caps require it
- host-side stop at exhaustion

This gives each run a bounded envelope.

Step limits (for example, "no more than 10 model/tool steps") are not inferred from budget consumption. An operator can configure standard caps such as `max_steps_remaining` or `max_tool_calls_remaining`, and the server can return them from `/decide` or with `ALLOW_WITH_CAPS`; the application must apply them. Combine a per-run workflow ledger with host-applied caps and separate authorization for layered run control.

### Why run budgets should usually be strict

Tenant budgets can be broad.

Workflow budgets can be product-shaped.

Run budgets should usually be narrow and safety-oriented.

They are your last line of defense against local execution instability.

## How the scopes work together

The real power comes from combining these scopes.

For example, a single action may need to satisfy all of the following:

- tenant still has available budget
- workflow is within its allowed envelope
- run has not exhausted its local execution cap

That means one reservation may be checked across multiple levels.

This is how Cycles turns budgeting into hierarchical governance rather than one flat counter.

## A practical example

Imagine a multi-tenant support platform.

A customer asks an agent to handle a refund issue.

The system may apply:

- **Tenant budget:** customer can consume up to 10,000 units per day
- **Workflow budget:** refund-assistant workflow can consume up to 2,000 units per day
- **Run budget:** this individual refund case can consume up to 250 units

Now suppose the workflow starts looping because the agent repeatedly tries tool calls.

The tenant budget might still have plenty of room.

The workflow budget might still be healthy too.

But the **run budget** can stop this one execution before it becomes a local incident.

That is why all three scopes matter.

## Which scope should enforce first?

In practice, the answer is usually:

**all relevant scopes should be checked before execution proceeds.**

But conceptually:

- tenant budgets protect platform economics
- workflow budgets protect product behavior
- run budgets protect execution safety

If you only have time to add one extra scope beyond tenant, add **run budgets** first.

That is often where the biggest operational safety gain appears.

## Recommended rollout order

If you are starting from scratch, use this rollout order:

### Phase 1: Tenant budgets

Start with account-level or customer-level boundaries.

This gives you immediate financial protection and multi-tenant isolation.

### Phase 2: Run budgets

Next, add hard limits for individual executions.

This protects against loops, runaway retries, and over-consumption inside otherwise healthy tenant budgets.

### Phase 3: Workflow budgets

Then introduce workflow-specific policy.

This helps product teams shape how different features are allowed to consume budget.

That sequence works well because it starts with the simplest boundary, then adds execution safety, then adds product nuance.

## Common mistakes

### Mistake 1: Only using tenant budgets

This makes the platform financially safer, but not necessarily operationally safer.

One runaway run can still do damage before the tenant budget is exhausted.

### Mistake 2: Making all workflows share the same envelope

Not all workflows have the same complexity, value, or risk.

Treating them the same usually leads to either over-permissive or overly restrictive policy.

### Mistake 3: Ignoring run-level ceilings

Run budgets are often the cleanest protection against accidental recursion and fan-out.

Skipping them is one of the fastest ways to leave a gap in the model.

### Mistake 4: Trying to model everything at once

Do not design a perfect hierarchy on day one.

Start with the scopes that map directly to real incidents.

For most teams, that means tenant + run first.

## How to think about policy design

A useful way to design budgets is to ask three questions:

### 1. What boundary are we protecting?

- platform economics?
- product feature behavior?
- individual execution safety?

### 2. What failure mode are we trying to prevent?

- customer over-consumption?
- expensive workflows?
- loops and retries?
- recursive tool use?
- noisy background jobs?

### 3. At what level should the stop happen?

- tenant
- workflow
- run

This keeps policy tied to operational reality instead of abstract hierarchy design.

## A strong default model

For many teams, a strong default looks like this:

- **Tenant budget** for daily or monthly usage boundaries
- **Workflow budget** for expensive or high-value process types
- **Run budget** for hard ceilings on individual executions

That gives you:

- platform isolation
- product-aware budget shaping
- local protection against runaway behavior

It is a practical, comprehensible starting model.

## Summary

Tenant, workflow, and run budgets are not redundant.

They each protect a different part of the system:

- **tenant budgets** protect the account boundary
- **workflow budgets** protect the process boundary
- **run budgets** protect the execution boundary

Cycles is most useful when these scopes are treated as complementary layers of governance rather than competing alternatives.

That is how teams move from simple usage caps to real autonomous execution control.

## Next steps

- [Tenants, Scopes, and Budgets](/how-to/understanding-tenants-scopes-and-budgets-in-cycles) — how tenants, scopes, and budgets work together as a unified model
- [Tenant Creation and Management](/how-to/tenant-creation-and-management-in-cycles) — create and configure tenants before setting up budgets
- [Budget Allocation and Management](/how-to/budget-allocation-and-management-in-cycles) — allocate and fund budgets at each scope level
- [API Key Management](/how-to/api-key-management-in-cycles) — create tenant-scoped API keys
- [Scope Derivation](/protocol/how-scope-derivation-works-in-cycles) — how subject fields map to hierarchical scope paths
- [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack) — set up the Cycles infrastructure from scratch
- Integrate with [Python](/quickstart/getting-started-with-the-python-client), [TypeScript](/quickstart/getting-started-with-the-typescript-client), or [Spring AI](/how-to/integrating-cycles-with-spring-ai)



# How to Think About Degradation Paths in Cycles: Deny, Downgrade, Disable, or Defer?

A budget boundary should not always mean a hard stop.

Sometimes the right answer is to deny execution immediately.

Sometimes the better answer is to continue in a cheaper, safer, or narrower mode.

That is where degradation paths matter.

As autonomous systems become more capable, control cannot be modeled as a simple binary between:

- allow everything
- block everything

Real systems often need a middle ground.

Cycles is designed to support bounded execution. In practice, that often means the most useful question is not only:

::: info
Should this action be denied?
:::

It is also:

::: info
If this action cannot proceed normally, what is the safest lower-cost behavior?
:::

This article explains how to think about degradation paths in Cycles, and when to choose:

- **deny**
- **downgrade**
- **disable**
- **defer**

## Why degradation matters

Hard enforcement is important.

But if every budget boundary becomes an immediate crash or rejection, teams often do one of two things:

- they make budgets too loose, because they are afraid of breaking production
- they avoid enforcement altogether, because the outcomes are too blunt

A good control layer needs more than a red light.

It needs graceful ways to reduce exposure when normal execution is no longer justified.

That is especially true in autonomous systems, where a single task may have several possible execution paths with different costs and different side-effect profiles.

## The core idea

When budget pressure appears, the system should not only decide whether work may continue.

It should also decide **how** work may continue.

That means moving from:

::: info
allow or deny
:::

to something closer to:

::: info
allow normally, allow in reduced mode, or stop
:::

This creates much better operational outcomes.

## The four main degradation paths

A useful starting model is:

- **deny** — stop the action
- **downgrade** — continue with a lower-cost or lower-risk path
- **disable** — remove a capability and continue without it
- **defer** — postpone execution until conditions improve

These are not the only possibilities, but they cover many real systems well.

## 1. Deny

Deny means the action does not proceed.

This is the strongest and clearest control.

Use deny when:

- the remaining budget is too low for safe execution
- the action is high risk or irreversible
- no safe cheaper path exists
- policy requires a hard stop
- continued execution would violate a strict tenant, workflow, or run boundary

### Examples

- block a payment-triggering workflow once run budget is exhausted
- stop a deployment action when environment budget is exceeded
- reject further tool calls when a recursive run crosses its ceiling
- prevent a tenant from exceeding its hard daily allocation

### When deny is best

Deny is best when:

- side effects are meaningful
- the action cannot be safely partially completed
- lower-cost alternatives would still be misleading or harmful
- the system needs a hard safety boundary

The main advantage of deny is clarity.

The main cost is user or workflow interruption.

## 2. Downgrade

Downgrade means the system continues, but with a lower-cost or lower-exposure execution path.

This is often the most useful degradation option for AI systems.

Use downgrade when:

- the task still has value in reduced form
- a cheaper model or path is available
- some quality loss is acceptable
- the system should preserve continuity while reducing exposure

### Examples

- switch from a larger model to a smaller model
- reduce context window size
- shorten generation length
- skip optional reasoning passes
- move from multi-step planning to direct response mode
- reduce retrieval breadth

### When downgrade is best

Downgrade is best when:

- the action is still useful at lower quality
- the main problem is cost rather than safety
- the system can preserve a reasonable user experience
- the product can tolerate graceful quality reduction

The main advantage of downgrade is continuity.

The main risk is silent quality loss if it is not well understood.

## 3. Disable

Disable means a specific capability is turned off while the broader workflow continues.

This is different from downgrade.

Downgrade changes the quality or cost of a path.  
Disable removes a capability from the path entirely.

Use disable when:

- one capability is disproportionately expensive
- one tool has high side-effect risk
- one feature is non-essential
- the system can still produce a meaningful outcome without that capability

### Examples

- disable web search when run budget is low
- disable ticket creation while still allowing read-only analysis
- disable external API calls but allow summarization
- disable file export while still returning an answer
- disable autonomous follow-up steps after budget pressure appears

### When disable is best

Disable is best when:

- the removed capability is optional
- the remaining workflow still has value
- the system should shrink its action surface under pressure
- the capability has higher risk than the rest of the flow

The main advantage of disable is that it reduces risk without always killing the whole experience.

The main tradeoff is reduced completeness.

## 4. Defer

Defer means the system does not execute now, but may execute later.

This is useful when immediate execution is not required and current conditions are unfavorable.

Use defer when:

- the action is important but not urgent
- budget may reset or refill later
- capacity is constrained temporarily
- the system should preserve intent without executing immediately

### Examples

- postpone a batch summarization job until the next budget window
- queue a non-urgent enrichment task for later execution
- defer expensive report generation until off-peak hours
- wait for tenant budget refill before resuming background work

### When defer is best

Defer is best when:

- user experience does not require immediate completion
- the value of the task remains later
- a later execution window is likely to be better
- you want to preserve work without forcing denial

The main advantage of defer is that it preserves intent without immediate exposure.

The main risk is operational complexity and backlog growth.

## How to choose between them

A useful way to choose the right degradation path is to ask four questions.

### 1. Is the action reversible?

If the action is irreversible or high-side-effect, prefer **deny** or **disable**.

Examples:
- payments
- writes
- ticket creation
- deployments

### 2. Is lower-quality output still valuable?

If yes, prefer **downgrade**.

Examples:
- summarization
- classification
- drafting
- general conversational responses

### 3. Is the expensive capability optional?

If yes, prefer **disable**.

Examples:
- web search
- optional tools
- non-critical enrichment
- follow-up actions

### 4. Is the task time-sensitive?

If not, prefer **defer**.

Examples:
- batch reporting
- background enrichment
- non-urgent analysis
- delayed follow-up jobs

These four questions usually make the policy direction clear.

## A practical mental model

You can think of the degradation options like this:

- **Deny** = stop the action
- **Downgrade** = do a cheaper version
- **Disable** = continue without a capability
- **Defer** = do it later

That is a useful framework for teams designing policy.

## Example: model call under budget pressure

Suppose a workflow wants to call a high-cost model but remaining budget is tight.

Possible paths:

- **Deny:** do not answer
- **Downgrade:** switch to a smaller model
- **Disable:** skip retrieval augmentation or tool use
- **Defer:** queue the task for later if it is non-urgent

In many conversational applications, downgrade is the best first option.

In some compliance or quality-critical workflows, deny may be more appropriate.

## Example: tool-heavy agent under run budget pressure

Suppose an agent has already consumed most of its run budget and wants to invoke another external tool.

Possible paths:

- **Deny:** stop the next tool call
- **Downgrade:** switch from multi-tool planning to direct answer mode
- **Disable:** turn off expensive tools and allow read-only reasoning
- **Defer:** suspend further work until another budget window

Here, disable is often strong because it narrows the action surface while still allowing bounded continuation.

## Example: tenant budget exhaustion

Suppose a tenant is near its daily limit.

Possible paths:

- **Deny:** block additional premium workflows
- **Downgrade:** route remaining requests to cheaper models
- **Disable:** turn off costly features for the rest of the window
- **Defer:** queue non-urgent tasks until budget resets

At the tenant level, multiple degradation paths may coexist by feature or action type.

## Deny is not failure

One important principle:

A good deny is often healthier than a bad continuation.

Teams sometimes avoid denial because it feels like a broken experience.

But unbounded execution is often worse.

A bounded stop with a clear policy reason is usually more operationally sound than silently allowing a system to exceed intended limits.

Degradation paths exist to make denial less blunt, not to eliminate it entirely.

## Degradation should be intentional, not accidental

Many systems already degrade, but accidentally.

For example:

- timeouts cause partial answers
- provider failures lead to implicit fallback
- missing tool responses cause odd behavior
- retry exhaustion produces brittle output

That is not the same as intentional degradation.

Cycles is most useful when degraded behavior is designed explicitly as part of policy.

That means the system knows:

- what should happen when reservation fails
- which cheaper alternatives are allowed
- which tools may be disabled
- what should be deferred
- when a hard stop is still the right answer

This turns budget pressure into a governed response, not a random one.

## A good rollout strategy

If you are introducing degradation paths for the first time, use this order:

### Phase 1: Hard deny for the highest-risk actions

Start with actions where continued execution is clearly unsafe or too costly.

### Phase 2: Downgrade for model-heavy paths

Add smaller-model or reduced-context alternatives where output remains useful.

### Phase 3: Disable optional expensive tools

Remove non-essential high-cost or high-side-effect capabilities when budget gets tight.

### Phase 4: Defer non-urgent work

Queue or postpone background actions that do not need immediate completion.

This sequence usually gives teams the best control with the least confusion.

## Common mistakes

### Mistake 1: Only thinking in binary allow/deny terms

This makes enforcement harder to adopt because it feels too brittle.

### Mistake 2: Downgrading silently without understanding quality impact

If output quality drops significantly, the product should understand and own that tradeoff.

### Mistake 3: Failing to distinguish expensive from dangerous

Some actions are mostly costly.  
Some are risky because of side effects.

The right degradation path may differ.

### Mistake 4: Deferring too much

Deferral is useful, but if overused it can create backlog, hidden debt, and delayed incidents.

### Mistake 5: Treating disable and downgrade as the same thing

They are related, but different.

Downgrade reduces the cost or quality of a path.  
Disable removes a capability entirely.

## Summary

Cycles is not only about saying no.

It is about making autonomous execution bounded and governable under real budget pressure.

That often means choosing among four main paths:

- **deny** when execution must stop
- **downgrade** when lower-cost execution is still valuable
- **disable** when a capability should be removed
- **defer** when work should happen later

These degradation paths help teams move from brittle enforcement to intentional control.

That is how runtime authority becomes usable in production.

## See also

- [Integrating Cycles with OpenClaw](/how-to/integrating-cycles-with-openclaw) — automatic model downgrade, tool access control, and budget-aware prompt injection via the OpenClaw plugin
- [Choosing the Right Overage Policy](/how-to/choosing-the-right-overage-policy) — how overage policies interact with degradation strategies
- [Common Budget Patterns](/how-to/common-budget-patterns) — budget structures that support graceful degradation

## Next steps

- [Error Handling Patterns](/how-to/error-handling-patterns-in-cycles-client-code) — implementing degradation in client code
- [Shadow Mode Rollout](/how-to/shadow-mode-in-cycles-how-to-roll-out-budget-enforcement-without-breaking-production) — test degradation paths before enforcing
- [Incident Patterns: Runaway Agents](/incidents/runaway-agents-tool-loops-and-budget-overruns-the-incidents-cycles-is-designed-to-prevent) — real-world scenarios where degradation prevents damage



Recipes for common tasks with Cycles, organized by what you're trying to do.

## Integrate Cycles into your app

- [Add Cycles with Claude or Codex](/how-to/add-cycles-with-claude-or-codex) — wire Cycles into an existing codebase using AI assistants.
- [Add Cycles to an existing application](/how-to/adding-cycles-to-an-existing-application) — manual integration steps.
- [Choose the right integration pattern](/how-to/choosing-the-right-integration-pattern) — decorator, middleware, manual, or framework-native.
- [Migrate from a custom rate limiter](/how-to/migrating-from-custom-rate-limiter-to-cycles)
- [Evaluate Cycles for an agent SaaS](/how-to/evaluate-cycles-for-agent-saas) — fit/no-fit framing and a 15-minute test.
- [Integrations overview](/how-to/integrations-overview) · [Ecosystem](/how-to/ecosystem)

### LLM providers

[OpenAI (Python)](/how-to/integrating-cycles-with-openai) · [OpenAI (TypeScript)](/how-to/integrating-cycles-with-openai-typescript) · [OpenAI (Rust / async-openai)](/how-to/integrating-cycles-with-async-openai) · [Anthropic (Python)](/how-to/integrating-cycles-with-anthropic) · [Anthropic (TypeScript)](/how-to/integrating-cycles-with-anthropic-typescript) · [AWS Bedrock](/how-to/integrating-cycles-with-aws-bedrock) · [Google Gemini](/how-to/integrating-cycles-with-google-gemini) · [Groq](/how-to/integrating-cycles-with-groq) · [Ollama / local LLMs](/how-to/integrating-cycles-with-ollama)

### Agent frameworks

[OpenAI Agents SDK](/how-to/integrating-cycles-with-openai-agents) · [LangChain (Python)](/how-to/integrating-cycles-with-langchain) · [LangChain.js](/how-to/integrating-cycles-with-langchain-js) · [LangGraph](/how-to/integrating-cycles-with-langgraph) · [LlamaIndex](/how-to/integrating-cycles-with-llamaindex) · [CrewAI](/how-to/integrating-cycles-with-crewai) · [AutoGen](/how-to/integrating-cycles-with-autogen) · [Pydantic AI](/how-to/integrating-cycles-with-pydantic-ai) · [any-agent](/how-to/integrating-cycles-with-anyagent) · [MCP](/how-to/integrating-cycles-with-mcp) · [OpenClaw](/how-to/integrating-cycles-with-openclaw)

### Web frameworks

[Vercel AI SDK](/how-to/integrating-cycles-with-vercel-ai-sdk) · [Next.js](/how-to/integrating-cycles-with-nextjs) · [Express](/how-to/integrating-cycles-with-express) · [FastAPI](/how-to/integrating-cycles-with-fastapi) · [Django](/how-to/integrating-cycles-with-django) · [Flask](/how-to/integrating-cycles-with-flask) · [Spring AI](/how-to/integrating-cycles-with-spring-ai) · [Rust](/how-to/integrating-cycles-with-rust)

## Design budget hierarchies

- [Choosing the right overage policy](/how-to/choosing-the-right-overage-policy) — reject, allow-if-available, or allow-with-overdraft.
- [Multi-tenant SaaS guide](/how-to/multi-tenant-saas-with-cycles)
- [Budget templates](/how-to/budget-templates) · [Common budget patterns](/how-to/common-budget-patterns)
- [Multi-agent shared workspace budgets](/how-to/multi-agent-shared-workspace-budget-patterns)
- [Cost estimation cheat sheet](/how-to/cost-estimation-cheat-sheet)
- [Budget allocation and management](/how-to/budget-allocation-and-management-in-cycles)
- [Tenant, workflow, and run budgets](/how-to/how-to-model-tenant-workflow-and-run-budgets-in-cycles)
- [Estimate exposure before execution](/how-to/how-to-estimate-exposure-before-execution-practical-reservation-strategies-for-cycles)
- [Assigning RISK_POINTS to agent tools](/how-to/assigning-risk-points-to-agent-tools)
- [Degradation paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer)
- [Budget control for LangChain agents](/how-to/how-to-add-budget-control-to-a-langchain-agent)
- [Shadow mode rollout](/how-to/shadow-mode-in-cycles-how-to-roll-out-budget-enforcement-without-breaking-production)

## Operate Cycles in production

- [Production operations guide](/how-to/production-operations-guide)
- [Upgrade Cycles safely](/how-to/upgrading-cycles) — preflight, rolling order, migration checks, rollback, and verification.
- [Redis backup, restore, and disaster recovery](/how-to/redis-backup-restore-disaster-recovery) — protect and recover the complete Cycles state store.
- [Monitoring and alerting](/how-to/monitoring-and-alerting) · [Observability setup](/how-to/observability-setup) · [Prometheus metrics reference](/how-to/prometheus-metrics-reference)
- [Security hardening](/how-to/security-hardening)
- [Tenant management](/how-to/tenant-creation-and-management-in-cycles) · [API key management](/how-to/api-key-management-in-cycles)
- [Tenants, scopes, and budgets](/how-to/understanding-tenants-scopes-and-budgets-in-cycles)
- [Rolling over billing periods](/how-to/rolling-over-billing-periods-with-reset-spent)
- [Bulk actions for tenants and webhooks](/how-to/using-bulk-actions-for-tenants-and-webhooks)
- [Force-releasing stuck reservations](/how-to/force-releasing-stuck-reservations-as-an-operator)
- [Searching admin list endpoints](/how-to/searching-and-sorting-admin-list-endpoints)
- [Webhook integrations](/how-to/webhook-integrations) · [Managing webhooks](/how-to/managing-webhooks)
- [Custom field resolvers](/how-to/custom-field-resolvers-in-cycles)
- [Programmatic client usage](/how-to/using-the-cycles-client-programmatically) · [Dashboard guide](/how-to/using-the-cycles-dashboard)
- [Client performance tuning](/how-to/client-performance-tuning)

## Handle errors and edge cases

- [Error handling patterns](/how-to/error-handling-patterns-in-cycles-client-code)
- Language-specific patterns: [Python](/how-to/error-handling-patterns-in-python) · [TypeScript](/how-to/error-handling-patterns-in-typescript) · [Rust](/how-to/error-handling-patterns-in-rust)
- [Handling streaming responses](/how-to/handling-streaming-responses-with-cycles)
- [Testing with Cycles](/how-to/testing-with-cycles)
- [Troubleshooting and FAQ](/how-to/troubleshooting-and-faq)

## Run the MCP server

- [Run the MCP server over HTTP](/how-to/running-the-mcp-server-over-http)

## Related

- [**Quickstart**](/quickstart/) — get started with a specific stack.
- [**Cycles Protocol**](/protocol/) — the open specification.
- [**How Cycles compares**](/concepts/comparisons) — vs LiteLLM, Helicone, rate limiters, provider caps, DIY wrappers.



# Integrating Cycles with Anthropic (TypeScript)

This guide shows how to guard Anthropic Messages API calls with Cycles budget reservations in TypeScript, including streaming support and per-tool-call budget tracking for agentic workflows.

For the Python version, see [Integrating with Anthropic (Python)](/how-to/integrating-cycles-with-anthropic).

## Prerequisites

- A running Cycles stack with a tenant, API key, and budget ([Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack))
- Node.js 20+

## Installation

```bash
npm install runcycles @anthropic-ai/sdk
```

```bash
export CYCLES_BASE_URL="http://localhost:7878"
export CYCLES_API_KEY="cyc_live_..."
export ANTHROPIC_API_KEY="sk-ant-..."
```

::: tip 60-Second Quick Start
```typescript
import Anthropic from "@anthropic-ai/sdk";
import { CyclesClient, CyclesConfig, withCycles } from "runcycles";

const cycles = new CyclesClient(CyclesConfig.fromEnv());
const anthropic = new Anthropic();

const ask = withCycles(
  {
    client: cycles,
    actionKind: "llm.completion",
    actionName: "claude-sonnet-4-6",
    estimate: () => 2_000_000,
    actual: (r: Anthropic.Message) =>
      r.usage.input_tokens * 300 + r.usage.output_tokens * 1_500,
  },
  async (prompt: string) => {
    return anthropic.messages.create({
      model: "claude-sonnet-4-6",
      max_tokens: 1024,
      messages: [{ role: "user", content: prompt }],
    });
  },
);

const response = await ask("What is budget authority?");
console.log(response.content[0].type === "text" ? response.content[0].text : "");
```
Budget is reserved before the call and committed with actual token cost after. If budget is exhausted, `BudgetExceededError` is thrown _before_ the Anthropic call is made.
:::

## Non-streaming calls with withCycles

Use the `withCycles` higher-order function to wrap Anthropic calls with automatic reserve → execute → commit:

```typescript
import Anthropic from "@anthropic-ai/sdk";
import {
  CyclesClient, CyclesConfig, withCycles,
  setDefaultClient, getCyclesContext, BudgetExceededError,
} from "runcycles";

const cyclesClient = new CyclesClient(CyclesConfig.fromEnv());
setDefaultClient(cyclesClient);

const anthropic = new Anthropic();

// Claude Sonnet 4.6 pricing (microcents per token; verified 2026-07-24)
const INPUT_PRICE = 300;     // $3.00 / 1M tokens
const OUTPUT_PRICE = 1_500;  // $15.00 / 1M tokens
const DEFAULT_MAX_TOKENS = 1024;

const sendMessage = withCycles(
  {
    client: cyclesClient,
    actionKind: "llm.completion",
    actionName: "claude-sonnet-4-6",
    estimate: (prompt: string) => {
      const inputTokens = Math.ceil(prompt.length / 4);
      return inputTokens * INPUT_PRICE + DEFAULT_MAX_TOKENS * OUTPUT_PRICE;
    },
    actual: (response: Anthropic.Message) => {
      return response.usage.input_tokens * INPUT_PRICE
        + response.usage.output_tokens * OUTPUT_PRICE;
    },
  },
  async (prompt: string) => {
    const ctx = getCyclesContext();

    // Respect budget caps
    let maxTokens = DEFAULT_MAX_TOKENS;
    if (ctx?.caps?.maxTokens) {
      maxTokens = Math.min(maxTokens, ctx.caps.maxTokens);
    }

    const response = await anthropic.messages.create({
      model: "claude-sonnet-4-6",
      max_tokens: maxTokens,
      messages: [{ role: "user", content: prompt }],
    });

    // Report metrics for observability
    if (ctx) {
      ctx.metrics = {
        tokensInput: response.usage.input_tokens,
        tokensOutput: response.usage.output_tokens,
        modelVersion: response.model,
      };
    }

    return response;
  },
);

try {
  const response = await sendMessage("Explain budget governance.");
  console.log(response.content[0].type === "text" ? response.content[0].text : "");
} catch (err) {
  if (err instanceof BudgetExceededError) {
    console.log("Budget exhausted.");
  } else {
    throw err;
  }
}
```

## Streaming with reserveForStream

For streaming responses, use `reserveForStream` to manage the reservation lifecycle:

```typescript
import Anthropic from "@anthropic-ai/sdk";
import {
  CyclesClient, CyclesConfig, reserveForStream, BudgetExceededError,
} from "runcycles";

const cyclesClient = new CyclesClient(CyclesConfig.fromEnv());
const anthropic = new Anthropic();

const INPUT_PRICE = 300;
const OUTPUT_PRICE = 1_500;

async function streamWithBudget(prompt: string) {
  const estimatedInputTokens = Math.ceil(prompt.length / 4);
  const estimate = estimatedInputTokens * INPUT_PRICE + 1024 * OUTPUT_PRICE;

  // 1. Reserve budget
  const handle = await reserveForStream({
    client: cyclesClient,
    estimate,
    unit: "USD_MICROCENTS",
    actionKind: "llm.completion",
    actionName: "claude-sonnet-4-6",
  });

  try {
    // Respect budget caps
    let maxTokens = 1024;
    if (handle.caps?.maxTokens) {
      maxTokens = Math.min(maxTokens, handle.caps.maxTokens);
    }

    // 2. Stream the response
    const stream = anthropic.messages.stream({
      model: "claude-sonnet-4-6",
      max_tokens: maxTokens,
      messages: [{ role: "user", content: prompt }],
    });

    for await (const event of stream) {
      if (
        event.type === "content_block_delta" &&
        event.delta.type === "text_delta"
      ) {
        process.stdout.write(event.delta.text);
      }
    }

    // 3. Commit actual usage from the final message
    const finalMessage = await stream.finalMessage();
    const actualCost =
      finalMessage.usage.input_tokens * INPUT_PRICE +
      finalMessage.usage.output_tokens * OUTPUT_PRICE;

    await handle.commit(actualCost, {
      tokensInput: finalMessage.usage.input_tokens,
      tokensOutput: finalMessage.usage.output_tokens,
      modelVersion: finalMessage.model,
    });
  } catch (err) {
    await handle.release("stream_error");
    throw err;
  }
}
```

## Per-tool-call budget tracking

When Claude uses tools, each LLM turn consumes tokens. Use `reserveForStream` to create a reservation per turn:

```typescript
import Anthropic from "@anthropic-ai/sdk";
import {
  CyclesClient, CyclesConfig, reserveForStream, BudgetExceededError,
} from "runcycles";

const cyclesClient = new CyclesClient(CyclesConfig.fromEnv());
const anthropic = new Anthropic();

const INPUT_PRICE = 300;
const OUTPUT_PRICE = 1_500;

async function chatWithTools(prompt: string): Promise<string> {
  const messages: Anthropic.MessageParam[] = [
    { role: "user", content: prompt },
  ];

  for (let turn = 1; turn <= 5; turn++) {
    // Reserve budget for this turn — a denial ends the conversation gracefully
    let handle;
    try {
      handle = await reserveForStream({
        client: cyclesClient,
        estimate: 2_000_000,
        unit: "USD_MICROCENTS",
        actionKind: "llm.completion",
        actionName: "claude-sonnet-4-6",
      });
    } catch (err) {
      if (err instanceof BudgetExceededError) {
        return "Budget exhausted mid-conversation.";
      }
      throw err;
    }

    try {
      const response = await anthropic.messages.create({
        model: "claude-sonnet-4-6",
        max_tokens: 1024,
        tools: TOOLS,
        messages,
      });

      // Commit actual cost
      const actualCost =
        response.usage.input_tokens * INPUT_PRICE +
        response.usage.output_tokens * OUTPUT_PRICE;

      await handle.commit(actualCost, {
        tokensInput: response.usage.input_tokens,
        tokensOutput: response.usage.output_tokens,
        modelVersion: response.model,
      });

      if (response.stop_reason === "end_turn") {
        const textBlock = response.content.find((b) => b.type === "text");
        return textBlock ? textBlock.text : "";
      }

      // Process tool calls and continue
      if (response.stop_reason === "tool_use") {
        messages.push({ role: "assistant", content: response.content });
        const toolResults: Anthropic.ToolResultBlockParam[] = [];
        for (const block of response.content) {
          if (block.type === "tool_use") {
            const result = await executeTool(block.name, block.input);
            toolResults.push({
              type: "tool_result",
              tool_use_id: block.id,
              content: result,
            });
          }
        }
        messages.push({ role: "user", content: toolResults });
      }
    } catch (err) {
      await handle.release("tool_call_error");
      throw err;
    }
  }

  return "Max turns reached.";
}
```

Each turn gets its own reservation, so the budget authority can deny mid-conversation if the agent is burning through budget too fast.

## Pricing reference

Adjust these constants for the model you use:

| Model | Input (microcents/token) | Output (microcents/token) |
|-------|--------------------------|---------------------------|
| Claude Haiku 4.5 | 100 | 500 |
| Claude Sonnet 4.6 | 300 | 1,500 |
| Claude Opus 4.8 | 500 | 2,500 |

Rates verified against Anthropic's pricing page on July 24, 2026. Recheck pricing before deploying, especially if you switch models, use prompt caching, or select regional inference.

## Key points

- **`withCycles` for non-streaming.** Wraps a single Anthropic call with automatic reserve → execute → commit.
- **`reserveForStream` for streaming.** Manages the reservation lifecycle with automatic heartbeat during the stream.
- **Token fields differ from OpenAI.** Anthropic uses `usage.input_tokens` / `usage.output_tokens` (not `prompt_tokens` / `completion_tokens`).
- **Per-turn reservations for tool use.** Each LLM turn in a tool-use loop gets its own reservation for fine-grained budget control.
- **Respect caps.** Check `handle.caps?.maxTokens` to honor budget authority limits.

## Full example

See [`examples/anthropic-sdk/`](https://github.com/runcycles/cycles-client-typescript/tree/main/examples/anthropic-sdk) for a complete, runnable example.

## Next steps

- [Integrating with Anthropic (Python)](/how-to/integrating-cycles-with-anthropic) — Python version of this guide
- [Handling Streaming Responses](/how-to/handling-streaming-responses-with-cycles) — streaming patterns in detail
- [Cost Estimation Cheat Sheet](/how-to/cost-estimation-cheat-sheet) — pricing reference for estimation
- [Error Handling in TypeScript](/how-to/error-handling-patterns-in-typescript) — handling budget errors
- [Production Operations Guide](/how-to/production-operations-guide) — running Cycles in production



# Integrating Cycles with Anthropic

This guide shows how to guard Anthropic Messages API calls with Cycles budget reservations, including per-tool-call budget tracking for agentic workflows.

## Prerequisites

```bash
pip install runcycles anthropic
```

```bash
export CYCLES_BASE_URL="http://localhost:7878"
export CYCLES_API_KEY="your-api-key"   # create via Admin Server — see note below
export CYCLES_TENANT="acme"
export ANTHROPIC_API_KEY="sk-ant-..."
```

> **Need an API key?** Create one via the Admin Server — see [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack#step-3-create-an-api-key) or [API Key Management](/how-to/api-key-management-in-cycles).

::: tip 60-Second Quick Start
```python
from anthropic import Anthropic
from runcycles import CyclesClient, CyclesConfig, cycles, set_default_client

set_default_client(CyclesClient(CyclesConfig.from_env()))

@cycles(estimate=2_000_000, action_kind="llm.completion", action_name="claude-sonnet-4-6")
def ask(prompt: str) -> str:
    return Anthropic().messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    ).content[0].text

print(ask("What is budget authority?"))
```
Every call is now budget-guarded. If the budget is exhausted, `BudgetExceededError` is raised _before_ the Anthropic call is made.

> **Note:** This quick start commits the estimate as actual spend. For accurate cost tracking, add an `actual` callback — see the decorator pattern below.
:::

## Simple decorator pattern

Use `@cycles` to wrap a single Anthropic call with automatic reserve → execute → commit:

```python
from anthropic import Anthropic
from runcycles import (
    CyclesConfig, CyclesClient, CyclesMetrics,
    cycles, get_cycles_context, set_default_client,
)

set_default_client(CyclesClient(CyclesConfig.from_env()))
anthropic_client = Anthropic()

PRICE_PER_INPUT_TOKEN = 300       # $3.00 / 1M tokens in microcents
PRICE_PER_OUTPUT_TOKEN = 1_500    # $15.00 / 1M tokens in microcents

@cycles(
    estimate=lambda prompt, **kw: (
        len(prompt.split()) * 2 * PRICE_PER_INPUT_TOKEN
        + kw.get("max_tokens", 1024) * PRICE_PER_OUTPUT_TOKEN
    ),
    actual=lambda result: (
        result["usage"]["input_tokens"] * PRICE_PER_INPUT_TOKEN
        + result["usage"]["output_tokens"] * PRICE_PER_OUTPUT_TOKEN
    ),
    action_kind="llm.completion",
    action_name="claude-sonnet-4-6",
    unit="USD_MICROCENTS",
    ttl_ms=60_000,
)
def send_message(prompt: str, max_tokens: int = 1024) -> dict:
    ctx = get_cycles_context()
    if ctx and ctx.has_caps() and ctx.caps.max_tokens:
        max_tokens = min(max_tokens, ctx.caps.max_tokens)

    response = anthropic_client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=max_tokens,
        messages=[{"role": "user", "content": prompt}],
    )

    if ctx:
        ctx.metrics = CyclesMetrics(
            tokens_input=response.usage.input_tokens,
            tokens_output=response.usage.output_tokens,
            model_version=response.model,
        )

    return {
        "content": response.content[0].text,
        "usage": {
            "input_tokens": response.usage.input_tokens,
            "output_tokens": response.usage.output_tokens,
        },
    }
```

## Per-tool-call budget tracking

When Claude uses tools, each LLM turn in the conversation consumes tokens. Use the programmatic client to create a separate reservation for each turn:

```python
import uuid
from runcycles import (
    CyclesClient, CyclesConfig, ReservationCreateRequest,
    CommitRequest, ReleaseRequest, Subject, Action, Amount,
    Unit, CyclesMetrics,
)

client = CyclesClient(CyclesConfig.from_env())
anthropic_client = Anthropic()

def chat_with_tools(prompt: str) -> str:
    messages = [{"role": "user", "content": prompt}]

    for turn in range(1, 6):  # max 5 turns
        key = str(uuid.uuid4())

        # Reserve budget for this turn
        res = client.create_reservation(ReservationCreateRequest(
            idempotency_key=key,
            subject=Subject(tenant="acme", agent="tool-agent"),
            action=Action(kind="llm.completion", name="claude-sonnet-4-6",
                          tags=[f"turn-{turn}"]),
            estimate=Amount(unit=Unit.USD_MICROCENTS, amount=2_000_000),
            ttl_ms=30_000,
        ))

        if not res.is_success:
            return "Budget exhausted — stopping."

        # Defensive: a conformant server returns 409 on live budget denial, but
        # dry-run responses (and lenient servers) return 200 with decision=DENY
        # and no reservation_id - check before using it
        if res.get_body_attribute("decision") == "DENY":
            return "Budget exhausted — stopping."

        reservation_id = res.get_body_attribute("reservation_id")

        # Call Claude with tools; release the reservation if the call fails
        try:
            response = anthropic_client.messages.create(
                model="claude-sonnet-4-6",
                max_tokens=1024,
                tools=TOOLS,
                messages=messages,
            )
        except Exception:
            client.release_reservation(reservation_id, ReleaseRequest(
                idempotency_key=f"release-{key}",
                reason="anthropic_call_failed",
            ))
            raise

        # Commit actual cost
        actual = (
            response.usage.input_tokens * PRICE_PER_INPUT_TOKEN
            + response.usage.output_tokens * PRICE_PER_OUTPUT_TOKEN
        )
        client.commit_reservation(reservation_id, CommitRequest(
            idempotency_key=f"commit-{key}",
            actual=Amount(unit=Unit.USD_MICROCENTS, amount=actual),
            metrics=CyclesMetrics(
                tokens_input=response.usage.input_tokens,
                tokens_output=response.usage.output_tokens,
                model_version=response.model,
                custom={"turn": turn},
            ),
        ))

        if response.stop_reason == "end_turn":
            return response.content[0].text

        # Process tool calls and continue
        if response.stop_reason == "tool_use":
            messages.append({"role": "assistant", "content": response.content})
            tool_results = []
            for block in response.content:
                if block.type == "tool_use":
                    result = execute_tool(block.name, block.input)
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": result,
                    })
            messages.append({"role": "user", "content": tool_results})

    return "Max turns reached."
```

Each turn gets its own reservation, so the budget authority can deny mid-conversation if the agent is burning through budget too fast.

## Pricing reference

Adjust these constants for the model you use:

| Model | Input (microcents/token) | Output (microcents/token) |
|-------|--------------------------|---------------------------|
| Claude Haiku 4.5 | 100 | 500 |
| Claude Sonnet 4.6 | 300 | 1,500 |
| Claude Opus 4.8 | 500 | 2,500 |

Rates verified against Anthropic's pricing page on July 24, 2026. Recheck pricing before deploying, especially if you switch models, use prompt caching, or select regional inference.

## Key points

- **Decorator for simple calls.** Use `@cycles` when you make a single API call and want automatic lifecycle management.
- **Programmatic client for multi-turn.** When tool use creates a loop of LLM calls, create a reservation per turn for fine-grained control.
- **Tag turns for observability.** Use `action.tags` (e.g., `["turn-1"]`) to distinguish costs across turns.
- **Custom metrics.** Use `CyclesMetrics.custom` to record tool-use metadata alongside standard token counts.
- **Always provide `actual`.** The `estimate` reserves budget before the call; the `actual` callback commits real cost from the response. Without `actual`, the estimate is committed as-is — overstating cost on short responses, understating on long ones.

## Full example

See [`examples/anthropic_integration.py`](https://github.com/runcycles/cycles-client-python/blob/main/examples/anthropic_integration.py) for a complete, runnable script.

## Next steps

- [Error Handling Patterns in Python](/how-to/error-handling-patterns-in-python) — handling budget errors in Python
- [Handling Streaming Responses](/how-to/handling-streaming-responses-with-cycles) — budget-managed streaming
- [Testing with Cycles](/how-to/testing-with-cycles) — testing budget-guarded code
- [Production Operations Guide](/how-to/production-operations-guide) — running Cycles in production
- [Anthropic example (TypeScript)](https://github.com/runcycles/cycles-client-typescript/tree/main/examples/anthropic-sdk) — runnable Anthropic SDK integration
- [Anthropic example (Python)](https://github.com/runcycles/cycles-client-python/blob/main/examples/anthropic_integration.py) — runnable Anthropic integration



# Integrating Cycles with AnyAgent

This guide shows how to add budget governance to [AnyAgent](https://mozilla-ai.github.io/any-agent/) workflows so that every LLM call and tool execution is cost-controlled, observable, and automatically stopped when budgets run out.

AnyAgent provides a unified interface for seven agent frameworks (OpenAI Agents, LangChain, LlamaIndex, Google, Agno, smolagents, TinyAgent). Because the Cycles callback hooks into AnyAgent's framework-agnostic callback system, a single integration covers all seven backends with no per-framework code.

## Prerequisites

```bash
pip install runcycles any-agent[all]
```

```bash
export CYCLES_BASE_URL="http://localhost:7878"
export CYCLES_API_KEY="your-api-key"   # create via Admin Server — see note below
export CYCLES_TENANT="acme"
export OPENAI_API_KEY="sk-..."
```

> **Need an API key?** Create one via the Admin Server — see [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack#step-3-create-an-api-key) or [API Key Management](/how-to/api-key-management-in-cycles).

::: tip 60-Second Quick Start
```python
from any_agent import AnyAgent, AgentConfig
from runcycles import CyclesClient, CyclesConfig

client = CyclesClient(CyclesConfig.from_env())

agent = AnyAgent.create(
    agent_framework="openai",
    agent_config=AgentConfig(
        model_id="openai:gpt-4o",
        instructions="You are a helpful assistant.",
        callbacks=[CyclesBudgetCallback(client=client, tenant="acme", agent="my-agent")],
    ),
)

trace = agent.run("What is budget authority?")
print(trace.final_output)
```
Every LLM call and tool execution is now budget-guarded. If the budget is exhausted, `BudgetExceeded` is raised _before_ the call is made. See the full `CyclesBudgetCallback` implementation below.
:::

## The callback approach

AnyAgent's callback system fires lifecycle hooks on every LLM call and tool execution. A custom `Callback` subclass can hook into `before_llm_call`, `after_llm_call`, `before_tool_execution`, and `after_tool_execution` to create and commit Cycles reservations:

```python
import uuid
from any_agent.callbacks.base import Callback
from any_agent import AgentCancel
from runcycles import (
    CyclesClient, CyclesConfig, ReservationCreateRequest, CommitRequest,
    ReleaseRequest, Subject, Action, Amount, Unit, CyclesMetrics,
    CyclesProtocolError,
)


class BudgetExceeded(AgentCancel):
    """Raised when Cycles denies a reservation due to budget exhaustion."""
    pass


class CyclesBudgetCallback(Callback):
    def __init__(
        self,
        client: CyclesClient | None = None,
        tenant: str = "default",
        workflow: str | None = None,
        agent: str | None = None,
        llm_estimate: int = 2_000_000,
        tool_estimate: int = 100_000,
        action_kind: str = "llm.completion",
        action_name: str = "gpt-4o",
    ):
        self.client = client or CyclesClient(CyclesConfig.from_env())
        self.tenant = tenant
        self.workflow = workflow
        self.agent = agent
        self.llm_estimate = llm_estimate
        self.tool_estimate = tool_estimate
        self.action_kind = action_kind
        self.action_name = action_name

    def _subject(self) -> Subject:
        return Subject(
            tenant=self.tenant,
            workflow=self.workflow,
            agent=self.agent,
        )

    def _reserve(self, context, kind: str, name: str, estimate: int):
        key = str(uuid.uuid4())
        res = self.client.create_reservation(ReservationCreateRequest(
            idempotency_key=key,
            subject=self._subject(),
            action=Action(kind=kind, name=name),
            estimate=Amount(unit=Unit.USD_MICROCENTS, amount=estimate),
            ttl_ms=60_000,
        ))

        if not res.is_success:
            error = res.get_error_response()
            if error and error.error == "BUDGET_EXCEEDED":
                raise BudgetExceeded(
                    error.message,
                )
            msg = error.message if error else (res.error_message or "Reservation failed")
            raise CyclesProtocolError(
                msg, status=res.status,
                error_code=error.error if error else None,
            )

        rid = res.get_body_attribute("reservation_id")
        context.shared.setdefault("_cycles_reservations", {})[rid] = key
        context.shared["_cycles_current_rid"] = rid
        return context

    def _commit(self, context, input_tokens: int = 0, output_tokens: int = 0):
        rid = context.shared.pop("_cycles_current_rid", None)
        reservations = context.shared.get("_cycles_reservations", {})
        key = reservations.pop(rid, None)
        if not rid or not key:
            return context

        self.client.commit_reservation(rid, CommitRequest(
            idempotency_key=f"commit-{key}",
            actual=Amount(unit=Unit.USD_MICROCENTS,
                          amount=input_tokens * 250 + output_tokens * 1_000),
            metrics=CyclesMetrics(
                tokens_input=input_tokens,
                tokens_output=output_tokens,
            ),
        ))
        return context

    def _release_current(self, context):
        rid = context.shared.pop("_cycles_current_rid", None)
        reservations = context.shared.get("_cycles_reservations", {})
        key = reservations.pop(rid, None)
        if rid and key:
            self.client.release_reservation(
                rid, ReleaseRequest(idempotency_key=f"release-{key}"),
            )
        return context

    def before_llm_call(self, context, *args, **kwargs):
        return self._reserve(context, self.action_kind, self.action_name, self.llm_estimate)

    def after_llm_call(self, context, *args, **kwargs):
        attrs = getattr(context.current_span, "attributes", None) or {}
        input_tokens = attrs.get("gen_ai.usage.input_tokens", 0)
        output_tokens = attrs.get("gen_ai.usage.output_tokens", 0)
        return self._commit(context, input_tokens, output_tokens)

    def before_tool_execution(self, context, *args, **kwargs):
        attrs = getattr(context.current_span, "attributes", None) or {}
        tool_name = attrs.get("gen_ai.tool.name", "unknown")
        return self._reserve(context, "tool.execution", tool_name, self.tool_estimate)

    def after_tool_execution(self, context, *args, **kwargs):
        return self._commit(context)
```

## Using the callback

> **Note:** Passing `callbacks=[...]` in `AgentConfig` replaces the default callbacks (including the console trace printer). To keep the default console output alongside budget governance, include `ConsolePrintSpan()`:
>
> ```python
> from any_agent.callbacks import ConsolePrintSpan
> callbacks=[CyclesBudgetCallback(...), ConsolePrintSpan()]
> ```

### Basic agent

```python
from any_agent import AnyAgent, AgentConfig
from runcycles import CyclesClient, CyclesConfig

client = CyclesClient(CyclesConfig.from_env())

callback = CyclesBudgetCallback(
    client=client,
    tenant="acme",
    agent="support-bot",
)

agent = AnyAgent.create(
    agent_framework="openai",
    agent_config=AgentConfig(
        model_id="openai:gpt-4o",
        instructions="You are a helpful assistant.",
        callbacks=[callback],
    ),
)

try:
    trace = agent.run("What's the weather in NYC?")
    print(trace.final_output)
except BudgetExceeded:
    print("Budget exhausted.")
```

### With tools

Every tool execution gets its own reservation:

```python
from any_agent import AnyAgent, AgentConfig
from any_agent.tools import search_web, visit_webpage

callback = CyclesBudgetCallback(
    client=client,
    tenant="acme",
    agent="research-agent",
    tool_estimate=200_000,
)

agent = AnyAgent.create(
    agent_framework="openai",
    agent_config=AgentConfig(
        model_id="openai:gpt-4o",
        instructions="Research topics using web search.",
        tools=[search_web, visit_webpage],
        callbacks=[callback],
    ),
)

try:
    trace = agent.run("Find the latest AI safety research papers")
    print(trace.final_output)
except BudgetExceeded:
    print("Agent stopped — budget exhausted.")
```

## Switching frameworks

The same callback works across all seven backends. Change the framework with a single parameter:

```python
# OpenAI Agents
agent = AnyAgent.create("openai", AgentConfig(
    model_id="openai:gpt-4o",
    callbacks=[callback],
    tools=[search_web],
))

# LangChain
agent = AnyAgent.create("langchain", AgentConfig(
    model_id="openai:gpt-4o",
    callbacks=[callback],
    tools=[search_web],
))

# Google (Gemini)
agent = AnyAgent.create("google", AgentConfig(
    model_id="google:gemini-2.0-flash",
    callbacks=[callback],
    tools=[search_web],
))
```

No changes to the callback — budget governance follows the agent across frameworks.

## Per-agent budget scoping

Use the `agent` parameter to scope budgets per agent role. This lets the budget authority set different limits for each agent:

```python
researcher_callback = CyclesBudgetCallback(
    client=client,
    tenant="acme",
    workflow="content-pipeline",
    agent="researcher",
    llm_estimate=3_000_000,
)

writer_callback = CyclesBudgetCallback(
    client=client,
    tenant="acme",
    workflow="content-pipeline",
    agent="writer",
    llm_estimate=2_000_000,
)

researcher = AnyAgent.create("openai", AgentConfig(
    model_id="openai:gpt-4o",
    instructions="Research topics thoroughly.",
    tools=[search_web],
    callbacks=[researcher_callback],
))

writer = AnyAgent.create("openai", AgentConfig(
    model_id="openai:gpt-4o",
    instructions="Write clear reports from research.",
    callbacks=[writer_callback],
))
```

This gives you a budget hierarchy: `tenant (acme)` > `workflow (content-pipeline)` > `agent (researcher / writer)`. Each agent can have its own budget limits set by the budget authority.

## Preflight budget check

Use `client.decide()` before creating the agent to check budget availability without consuming tokens:

```python
import uuid
from runcycles import DecisionRequest, Subject, Action, Amount, Unit

response = client.decide(DecisionRequest(
    idempotency_key=f"decide-{uuid.uuid4()}",
    subject=Subject(tenant="acme", agent="support-bot"),
    action=Action(kind="llm.completion", name="gpt-4o"),
    estimate=Amount(unit=Unit.USD_MICROCENTS, amount=5_000_000),
))

if response.is_success:
    decision = response.get_body_attribute("decision")
    if decision == "DENY":
        print("Budget insufficient — skipping agent run.")
    else:
        trace = agent.run("Handle this support ticket")
```

## Async usage

AnyAgent supports async throughout. The callback hooks work identically:

```python
agent = await AnyAgent.create_async(
    agent_framework="openai",
    agent_config=AgentConfig(
        model_id="openai:gpt-4o",
        instructions="You are a helpful assistant.",
        callbacks=[CyclesBudgetCallback(tenant="acme", agent="async-agent")],
    ),
)

try:
    trace = await agent.run_async("What is budget authority?")
    print(trace.final_output)
except BudgetExceeded:
    print("Budget exhausted.")
```

## Error handling

When budget is denied, `BudgetExceeded` (a subclass of `AgentCancel`) propagates up from the callback. AnyAgent preserves the partial trace:

```python
from any_agent import AgentRunError

try:
    trace = agent.run("Process this request")
except BudgetExceeded as e:
    print(f"Budget denied: {e}")
    print(f"Partial trace: {e.trace}")  # trace up to cancellation point
except AgentRunError as e:
    print(f"Unexpected error: {e.original_exception}")
    print(f"Trace: {e.trace}")
```

For pipelines where partial completion is acceptable, run agents sequentially and handle errors at each stage:

```python
try:
    research_trace = researcher.run("quantum computing")
    research = research_trace.final_output
except BudgetExceeded:
    research = cached_research.get("quantum computing", "No data available.")

try:
    report_trace = writer.run(f"Write a report based on: {research}")
    report = report_trace.final_output
except BudgetExceeded:
    report = f"Raw research (report generation skipped):\n{research}"
```

See [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) for patterns like queueing, model downgrade, and caching.

## Post-run cost analysis

AnyAgent's `AgentTrace` provides token and cost data after execution. Combine this with Cycles for both enforcement and observability:

```python
trace = agent.run("Summarize this document")

# AnyAgent's built-in cost tracking
print(f"Duration: {trace.duration}")
print(f"Tokens: {trace.tokens}")
print(f"Cost: {trace.cost}")

# Cycles budget tracking
response = client.get_balances(tenant="acme")
if response.is_success:
    for balance in response.body.get("balances", []):
        print(f"Scope: {balance['scope']}, remaining: {balance['remaining']}")
```

## Key points

- **One callback covers all frameworks.** The `CyclesBudgetCallback` works identically across all seven AnyAgent backends — no per-framework code needed.
- **LLM calls and tool executions are both guarded.** `before_llm_call` and `before_tool_execution` each create a reservation; `after_*` hooks commit actual cost.
- **`AgentCancel` stops cleanly.** `BudgetExceeded` extends `AgentCancel`, so AnyAgent preserves the partial trace and stops the agent without wrapping the error.
- **Per-agent scoping with subject hierarchy.** Use `tenant`, `workflow`, and `agent` to mirror your agent topology in Cycles budget paths.
- **Preflight checks with `client.decide()`.** Check budget availability before creating or running the agent to avoid wasting resources.

## Next steps

- [Error Handling Patterns in Python](/how-to/error-handling-patterns-in-python) — handling budget errors in Python
- [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) — strategies for graceful degradation
- [Testing with Cycles](/how-to/testing-with-cycles) — testing budget-guarded code
- [Production Operations Guide](/how-to/production-operations-guide) — running Cycles in production
- [Integrating with OpenAI](/how-to/integrating-cycles-with-openai) — budget governance for direct OpenAI calls
- [Integrating with LangChain](/how-to/integrating-cycles-with-langchain) — budget governance for LangChain apps



# Integrate Cycles with async-openai (Rust)

The [Rust quickstart](/quickstart/getting-started-with-the-rust-client) and [Rust integration guide](/how-to/integrating-cycles-with-rust) both use a `call_llm()` placeholder where a real OpenAI call should go. This page fills that gap: it shows how `runcycles` composes with [`async-openai`](https://crates.io/crates/async-openai) (a widely used Rust client for the OpenAI API) for chat completions, streaming, and token-accurate commits.

The same lifecycle composes against other Rust LLM clients (Anthropic, Bedrock, local LLMs via Ollama) — the reserve-commit shape doesn't change. See the brief [Other Rust LLM clients](#other-rust-llm-clients) note at the bottom.

## What you get

- `with_cycles()` wrapping a real OpenAI call, with `prompt_tokens + completion_tokens` flowing through to the commit
- A `ReservationGuard` pattern for streaming chat completions where token counts are only known at the end of the stream
- Error-aware patterns using `ReservationGuard` that preserve typed `OpenAIError` for the caller (`with_cycles()` wraps closure errors as `Error::Validation` and loses the original type)
- Token-to-USD conversion at commit time for spend-denominated budgets

**Loud-failure stance.** All four examples on this page error out on missing `usage` or missing `content` rather than silently committing zero. The examples that read `caps.max_tokens` (the ALLOW_WITH_CAPS example, the streaming example, and the error-aware example) additionally error on non-positive cap values rather than sending `max_completion_tokens=0` to OpenAI. The basic example deliberately ignores `ctx.caps` to keep the minimum-viable composition compact — production code should follow the capped pattern. This matches the shipped [`examples/async_openai_completion.rs`](https://github.com/runcycles/cycles-client-rust/blob/main/examples/async_openai_completion.rs) in the runcycles crate. Production code that prefers a fallback (e.g. commit the reservation estimate on missing usage) should opt into that fallback explicitly — the default in a teaching example should not be silent under-billing.

## Cargo.toml

```toml
[dependencies]
runcycles = "0.3"
async-openai = { version = "0.38", default-features = false, features = ["chat-completion", "rustls"] }
tokio = { version = "1", features = ["full"] }
futures = "0.3"                # for stream consumption
thiserror = "2"                # for the error-aware section
```

`async-openai` 0.31+ splits its surface behind per-API features — the `chat-completion` feature is what makes `Client` and the chat-completion types available. The 0.30.x line bundled everything by default; if you're upgrading from there, the example uses `async_openai::types::chat::` paths (the chat types moved out of the top-level `types::` module in 0.31). The 0.30.x line also pulled `backoff` transitively, which has been replaced with `tower` in 0.31+ — worth the version bump for the cleaner dependency tree alone.

## The basic pattern: with_cycles + chat completions

```rust
use async_openai::{
    Client,
    types::chat::{CreateChatCompletionRequestArgs, ChatCompletionRequestUserMessageArgs},
};
use runcycles::{
    CyclesClient, with_cycles, WithCyclesConfig,
    models::{Amount, Subject, CyclesMetrics},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let cycles = CyclesClient::builder("cyc_live_...", "http://localhost:7878")
        .tenant("acme-corp")
        .build();
    let openai = Client::new();

    let prompt = "Summarize the runcycles crate in one sentence.";

    let reply = with_cycles(
        &cycles,
        WithCyclesConfig::new(Amount::tokens(1_500))
            .action("llm.completion", "gpt-4o-mini")
            .subject(Subject {
                tenant: Some("acme-corp".into()),
                ..Default::default()
            }),
        |_ctx| async move {
            let request = CreateChatCompletionRequestArgs::default()
                .model("gpt-4o-mini")
                // max_completion_tokens is the current field; max_tokens is
                // deprecated upstream for chat completions.
                .max_completion_tokens(800u32)
                .messages([ChatCompletionRequestUserMessageArgs::default()
                    .content(prompt)
                    .build()?
                    .into()])
                .build()?;

            let response = openai.chat().create(request).await?;

            // Loud-failure stance: a successful HTTP response with no choices
            // / no content is a malformed result. Surfacing it as `Err` lets
            // `with_cycles` release the reservation rather than commit on an
            // empty reply.
            let text = response
                .choices
                .first()
                .and_then(|c| c.message.content.clone())
                .ok_or("OpenAI response had no message content")?;

            // Same stance for missing usage: committing zero tokens against a
            // successful-looking call silently under-bills the budget. Error
            // out and let the caller decide whether to fall back.
            let usage = response
                .usage
                .ok_or("OpenAI response omitted usage — refusing to commit a guessed amount")?;
            let actual = i64::from(usage.total_tokens);

            Ok((text, Amount::tokens(actual)))
        },
    )
    .await?;

    println!("{reply}");
    Ok(())
}
```

### What's happening

| Step | What runs | What is recorded |
|---|---|---|
| Before the closure | Cycles reserves `1_500` [tokens](/glossary#token) against the request subject | Reservation created, decision evaluated |
| Inside the closure | `openai.chat().create(request)` issues the actual API call | OpenAI bills your account for the real usage |
| Return value | `(text, Amount::tokens(actual_total))` | The actual `total_tokens` becomes the commit amount |
| After the closure | Cycles commits `actual` tokens, releases the unused reservation | Final spend recorded; the reservation lifecycle closes |

If `openai.chat().create()` returns `Err`, the closure returns `Err` and the reservation is released — no commit, no false spend record.

## Capping max_tokens from `ALLOW_WITH_CAPS`

When Cycles returns `ALLOW_WITH_CAPS`, the `GuardContext` carries the server's cap suggestions. Apply them to the OpenAI request before issuing it:

```rust
let reply = with_cycles(
    &cycles,
    WithCyclesConfig::new(Amount::tokens(1_500))
        .action("llm.completion", "gpt-4o-mini")
        .subject(Subject { tenant: Some("acme-corp".into()), ..Default::default() }),
    |ctx| async move {
        // Default ceiling; override if Cycles capped lower. A non-positive
        // cap is treated as an explicit refusal — sending max_completion_tokens=0
        // would charge the request for zero output, which is never the intent.
        let mut max_tokens: u32 = 800;
        if let Some(caps) = &ctx.caps {
            if let Some(cap) = caps.max_tokens {
                let cap_u32 = u32::try_from(cap)
                    .map_err(|_| "caps.max_tokens is negative — refusing to call OpenAI")?;
                if cap_u32 == 0 {
                    return Err("caps.max_tokens is 0 — refusing to call OpenAI".into());
                }
                max_tokens = cap_u32.min(max_tokens);
            }
        }

        let request = CreateChatCompletionRequestArgs::default()
            .model("gpt-4o-mini")
            .max_completion_tokens(max_tokens)
            .messages([ChatCompletionRequestUserMessageArgs::default()
                .content(prompt)
                .build()?
                .into()])
            .build()?;

        let response = openai.chat().create(request).await?;
        let text = response.choices.first()
            .and_then(|c| c.message.content.clone())
            .ok_or("OpenAI response had no message content")?;
        let usage = response.usage
            .ok_or("OpenAI response omitted usage")?;
        Ok((text, Amount::tokens(i64::from(usage.total_tokens))))
    },
).await?;
```

`caps.tool_allowlist` and `caps.tool_denylist` follow the same shape — if you wire OpenAI's function-calling tools, use those caps to filter your tool list before passing it to the request builder. See [Caps and the Three-Way Decision Model](/protocol/caps-and-the-three-way-decision-model-in-cycles) for the full cap surface.

## Streaming: ReservationGuard with stream consumption

Streaming chat completions return tokens one chunk at a time. The total token count is only known after the stream ends, which means `with_cycles()` (which expects the closure to return both the value and the actual cost in one go) is not the right primitive. Use a `ReservationGuard` instead.

OpenAI's streaming endpoint emits a final `usage` chunk only when `stream_options.include_usage` is set on the request. Set it explicitly:

```rust
use async_openai::{
    Client,
    types::chat::{
        CreateChatCompletionRequestArgs,
        ChatCompletionRequestUserMessageArgs,
        ChatCompletionStreamOptions,
    },
};
use futures::StreamExt;
use runcycles::{
    CyclesClient,
    models::{
        Amount, Subject, Action, ReservationCreateRequest, CommitRequest, CyclesMetrics,
    },
};

let openai = Client::new();

let guard = cycles.reserve(
    ReservationCreateRequest::builder()
        .subject(Subject { tenant: Some("acme-corp".into()), ..Default::default() })
        .action(Action::new("llm.completion", "gpt-4o-mini"))
        .estimate(Amount::tokens(2_000))
        .ttl_ms(60_000_u64)
        .build()
).await?;

// Apply caps before building the request. Non-positive caps are an explicit
// refusal — release the guard and bail rather than send max_completion_tokens=0.
// Note `let _ = ... .await` on release: if the release itself errors (rare —
// network failure between the agent and the Cycles server), the caller still
// sees the original zero-cap error rather than the release error swallowing
// it.
let mut max_tokens: u32 = 1_500;
if let Some(caps) = guard.caps() {
    if let Some(cap) = caps.max_tokens {
        let cap_u32 = u32::try_from(cap)
            .map_err(|_| "caps.max_tokens is negative — refusing to call OpenAI")?;
        if cap_u32 == 0 {
            let _ = guard.release("caps.max_tokens is 0".to_string()).await;
            return Err("caps.max_tokens is 0 — refusing to call OpenAI".into());
        }
        max_tokens = cap_u32.min(max_tokens);
    }
}

let request = CreateChatCompletionRequestArgs::default()
    .model("gpt-4o-mini")
    .max_completion_tokens(max_tokens)
    .messages([ChatCompletionRequestUserMessageArgs::default()
        .content(prompt)
        .build()?
        .into()])
    .stream(true)
    // Required for the stream to emit a final usage chunk. The struct's
    // fields are `Option<bool>` in async-openai 0.38.x — `include_obfuscation`
    // is set to `None` to keep the upstream default.
    .stream_options(ChatCompletionStreamOptions {
        include_usage: Some(true),
        include_obfuscation: None,
    })
    .build()?;

let mut stream = openai.chat().create_stream(request).await?;

let mut full_text = String::new();
let mut final_usage_tokens: i64 = 0;

while let Some(chunk_result) = stream.next().await {
    let chunk = chunk_result?;
    for choice in chunk.choices {
        if let Some(content) = choice.delta.content {
            full_text.push_str(&content);
        }
    }
    // The final chunk carries usage when include_usage was set.
    if let Some(usage) = chunk.usage {
        final_usage_tokens = i64::from(usage.total_tokens);
    }
}

// Two edge cases at end-of-stream:
//
//   - `full_text` is empty: the stream produced no content chunks. Treat as
//     a malformed result and release the guard rather than commit on a
//     zero-output response.
//   - `final_usage_tokens` is zero: the stream completed but the provider
//     didn't honor `include_usage`. Some OpenAI-compatible servers (Ollama,
//     vLLM, certain LiteLLM configs) silently drop the usage chunk. Either
//     estimate locally with a tokenizer, or release and error.
//
// The example below takes the loud path (release + error) to match the
// non-streaming sections' stance. For production code that prefers a
// fallback, plug in the `tiktoken-rs` crate's `o200k_base()` encoder and
// commit the estimate — see the snippet at the end of this section.
if full_text.is_empty() {
    let _ = guard.release("openai_stream_no_content".to_string()).await;
    return Err("OpenAI stream produced no content".into());
}
if final_usage_tokens == 0 {
    let _ = guard.release("openai_stream_no_usage".to_string()).await;
    return Err(
        "OpenAI stream omitted usage — set stream_options.include_usage or estimate locally".into(),
    );
}

guard.commit(
    CommitRequest::builder()
        .actual(Amount::tokens(final_usage_tokens))
        .metrics(CyclesMetrics {
            tokens_output: Some(final_usage_tokens),
            ..Default::default()
        })
        .build()
).await?;
```

### Why the guard, not `with_cycles`

`with_cycles()` evaluates the closure to a `(value, actual_cost)` tuple in one synchronous return. Streaming requires you to drive the stream to completion (which can take seconds), then commit the total. The guard exposes that lifecycle as two explicit steps — reserve before the stream begins, commit after it ends.

If the stream errors midway (network failure, rate limit, content policy violation), call `guard.release(...).await?` — the reservation is returned to the pool with a reason code. The guard's `Drop` implementation provides best-effort release on panic / early `?` return, but explicit release with a reason code is preferred for clean audit records.

### Optional: tokenizer fallback for missing-usage chunks

If the loud-failure path on missing usage is too pessimistic for your deployment — for instance, you're routing through an OpenAI-compatible proxy that doesn't honor `include_usage` and you can't change the proxy — plug in a real tokenizer instead of erroring out. The `tiktoken-rs` crate's `o200k_base` encoder matches the tokenizer used by gpt-4o-family models:

```rust
// Add to Cargo.toml: tiktoken-rs = "0.11"   (check crates.io for current)
use tiktoken_rs::o200k_base;

fn estimate_tokens(prompt: &str, output: &str) -> Result<i64, Box<dyn std::error::Error + Send + Sync>> {
    let bpe = o200k_base()?;
    let input = i64::try_from(bpe.encode_with_special_tokens(prompt).len())?;
    let out = i64::try_from(bpe.encode_with_special_tokens(output).len())?;
    Ok(input + out)
}
```

Then commit `estimate_tokens(&prompt, &full_text)` instead of releasing the guard on the missing-usage branch. The estimate will be approximate — it doesn't account for system prompts, tool definitions, or the model's actual tokenization of formatting tokens — but it beats committing zero.

## Error handling: preserving the OpenAI error type

`async-openai` returns `OpenAIError`; Cycles returns `runcycles::Error`. Callers usually want to act on these differently:

- **OpenAI errors** — rate-limit retries with backoff, model fallback (gpt-4o → gpt-4o-mini), prompt resubmission.
- **Cycles errors** — [graceful degradation](/glossary#graceful-degradation) to a smaller model, deferred response, "budget exhausted" UX.

`with_cycles()` is *not* the right primitive for error-aware flows. Its closure must return `Result<(T, Amount), Box<dyn std::error::Error + Send + Sync>>`, and any closure error is wrapped as `runcycles::Error::Validation(format!("guarded function failed: {e}"))`. The original typed error is stringified into the message and lost — the caller cannot recover it.

For flows that need to act on the typed `OpenAIError`, use `ReservationGuard` and keep the error visible to the caller:

```rust
use async_openai::{
    Client,
    error::OpenAIError,
    types::chat::{CreateChatCompletionRequestArgs, ChatCompletionRequestUserMessageArgs},
};
use runcycles::{
    CyclesClient, Error as CyclesError,
    models::{Amount, Subject, Action, ReservationCreateRequest, CommitRequest},
};

#[derive(Debug, thiserror::Error)]
enum CompletionError {
    #[error(transparent)] OpenAi(#[from] OpenAIError),
    #[error(transparent)] Cycles(#[from] CyclesError),
}

async fn run_completion(
    cycles: &CyclesClient,
    openai: &Client<async_openai::config::OpenAIConfig>,
    prompt: &str,
) -> Result<String, CompletionError> {
    let guard = cycles.reserve(
        ReservationCreateRequest::builder()
            .subject(Subject { tenant: Some("acme-corp".into()), ..Default::default() })
            .action(Action::new("llm.completion", "gpt-4o-mini"))
            .estimate(Amount::tokens(1_500))
            .build()
    ).await?;

    // Same cap handling as the other examples: non-positive caps release the
    // guard and surface a typed Cycles error rather than send
    // max_completion_tokens=0 to OpenAI.
    let mut max_tokens: u32 = 800;
    if let Some(caps) = guard.caps() {
        if let Some(cap) = caps.max_tokens {
            let cap_u32 = u32::try_from(cap).map_err(|_| {
                CompletionError::Cycles(CyclesError::Validation(
                    "caps.max_tokens is negative".into(),
                ))
            })?;
            if cap_u32 == 0 {
                let _ = guard.release("caps.max_tokens is 0".to_string()).await;
                return Err(CompletionError::Cycles(CyclesError::Validation(
                    "caps.max_tokens is 0".into(),
                )));
            }
            max_tokens = cap_u32.min(max_tokens);
        }
    }

    let request = CreateChatCompletionRequestArgs::default()
        .model("gpt-4o-mini")
        .max_completion_tokens(max_tokens)
        .messages([ChatCompletionRequestUserMessageArgs::default()
            .content(prompt)
            .build()?
            .into()])
        .build()?;

    let response = match openai.chat().create(request).await {
        Ok(r) => r,
        Err(e) => {
            // Release the reservation with a reason; preserve the typed OpenAI error
            let _ = guard.release(format!("openai_error: {e}")).await;
            return Err(e.into()); // OpenAIError flows to the caller
        }
    };

    // Loud failure on malformed-but-successful responses: missing content or
    // missing usage releases the reservation and surfaces as a typed error,
    // rather than committing zero and silently under-billing.
    let text = match response.choices.first().and_then(|c| c.message.content.clone()) {
        Some(t) => t,
        None => {
            let _ = guard.release("openai_no_content".to_string()).await;
            return Err(CompletionError::Cycles(CyclesError::Validation(
                "OpenAI response had no message content".into(),
            )));
        }
    };
    let usage = match response.usage {
        Some(u) => u,
        None => {
            let _ = guard.release("openai_no_usage".to_string()).await;
            return Err(CompletionError::Cycles(CyclesError::Validation(
                "OpenAI response omitted usage".into(),
            )));
        }
    };

    guard.commit(
        CommitRequest::builder()
            .actual(Amount::tokens(i64::from(usage.total_tokens)))
            .build()
    ).await?;

    Ok(text)
}
```

At the call site, the typed branches are now available:

```rust
match run_completion(&cycles, &openai, prompt).await {
    Ok(text) => println!("{text}"),
    Err(CompletionError::OpenAi(_e)) => {
        // backoff / retry / fallback model
    }
    Err(CompletionError::Cycles(CyclesError::BudgetExceeded { retry_after, .. })) => {
        // graceful degradation — defer, downsize model, return cached response
        let _ = retry_after;
    }
    Err(CompletionError::Cycles(other)) => {
        // log and surface
        eprintln!("cycles error: {other}");
    }
}
```

Use `with_cycles()` when the caller doesn't need to distinguish the underlying error type — for fire-and-forget background tasks, scripts, or higher-level orchestrators that uniformly retry on any failure. Switch to `ReservationGuard` whenever the caller needs to branch on the actual error.

The Cycles error types and their convenience methods (`is_retryable`, `is_budget_exceeded`, `retry_after`) are covered in [Error Handling in Rust](/how-to/error-handling-patterns-in-rust).

## Token-to-USD: when your budget is denominated in dollars, not tokens

If the budget unit is `USD_MICROCENTS` rather than `TOKENS`, convert from the response usage at commit time:

```rust
fn tokens_to_microcents(prompt_tokens: u32, completion_tokens: u32, model: &str) -> u64 {
    // Rates expressed as microcents per million tokens (1 cent = 1_000_000 microcents).
    // The numbers below illustrate; pin yours to the provider's current pricing
    // page and bump them as a release task — model rates change.
    let (input_per_million_microcents, output_per_million_microcents) = match model {
        "gpt-4o-mini" => (15_000_000, 60_000_000),     // illustrative: $0.15 / $0.60
        "gpt-4o"      => (250_000_000, 1_000_000_000), // illustrative: $2.50 / $10.00
        _             => (15_000_000, 60_000_000),
    };
    let input  = (prompt_tokens as u64)     * input_per_million_microcents  / 1_000_000;
    let output = (completion_tokens as u64) * output_per_million_microcents / 1_000_000;
    input + output
}

// Inside the with_cycles closure:
let usage = response.usage
    .ok_or("OpenAI response omitted usage — refusing to commit a guessed amount")?;
let microcents = tokens_to_microcents(usage.prompt_tokens, usage.completion_tokens, "gpt-4o-mini");
let amount = i64::try_from(microcents)
    .map_err(|_| "microcents overflow when converting to i64")?;
Ok((text, Amount::usd_microcents(amount)))
```

Keeping the rate table in one helper makes provider rate changes a single-edit fix. For multi-provider deployments, hoist it to your shared `costs` module.

For the canonical breakdown of provider rates and the cost-estimation patterns used elsewhere in the docs, see [Cost Estimation Cheat Sheet](/how-to/cost-estimation-cheat-sheet).

## Other Rust LLM clients

The reserve-commit shape is the same for any Rust LLM client. The four things you need to adapt to a new provider:

1. **The request builder type** — `CreateChatCompletionRequestArgs` for async-openai, `MessageCreateBuilder` / `MessageCreateParams` for Anthropic's `anthropic-sdk-rust`, the provider-specific equivalent elsewhere.
2. **The call method** — `client.chat().create(req)` for async-openai; consult the provider crate's docs for the equivalent.
3. **The response usage extraction** — `response.usage.ok_or(...)?` then `i64::from(usage.total_tokens)` for async-openai (loud failure on missing usage, no `as` cast). Anthropic returns `input_tokens` + `output_tokens` separately on its response usage object; the same `ok_or(...)?` / `i64::from(...)` pattern applies, you just sum the two fields.
4. **The model name in the action label** — `.action("llm.completion", "claude-3-5-sonnet-20241022")` rather than `"gpt-4o-mini"`.

Pin to the specific crate version you're using and verify each of those four points against its current docs before copy-pasting. The Rust Anthropic ecosystem in particular has churn across crate names and major versions; the reserve-commit lifecycle is unchanged, but the provider-side type paths are not portable.

The [`Error Handling in Rust`](/how-to/error-handling-patterns-in-rust) patterns apply to all providers — the typed `OpenAIError` branch above becomes a typed `AnthropicError` (or equivalent) branch for the other crate.

## Common gotchas

1. **Streaming without `include_usage` reports zero tokens.** OpenAI's official streaming endpoint emits usage only when `stream_options.include_usage` is set on the request. Without it, you'll commit zero tokens and the budget will not reflect actual spend. Set the option, and have a tokenizer fallback for OpenAI-compatible providers that don't honor it.

2. **`response.usage` is `Option`.** Some compatible servers (Ollama, vLLM, certain LiteLLM configs) don't return usage. For **non-streaming** calls, the cleanest pattern is loud failure — return `Err`, let `with_cycles` release the reservation, surface the issue to the caller (the examples above follow this stance, matching the shipped `cycles-client-rust/examples/async_openai_completion.rs`). Streaming is the genuine exception: you've already consumed the stream so re-issuing is expensive, and a tokenizer estimate beats committing zero.

3. **`response.choices[0].message.content` can be `None`** when the model returns only a tool-call, a refusal, or finishes with `length` on a malformed setup. Treat that as a malformed result (fail loud and release) rather than committing on an empty reply.

4. **Don't include the OpenAI API key in the Cycles reservation metadata.** Cycles records actions, not credentials. If you're tagging the reservation with provider info, use the action name (`gpt-4o-mini`) — never the key.

5. **Mismatched async runtimes.** `async-openai` uses `tokio`; the blocking `runcycles` variant requires not being inside a Tokio runtime. Pick one — for most LLM workloads, the async client is correct.

6. **`as u32` / `as i64` on values you got from elsewhere.** `cap as u32` silently wraps on a negative `cap.max_tokens`; `microcents as i64` silently wraps on overflow. Use `u32::try_from(...)` / `i64::try_from(...)` and surface a typed error instead.

## Next steps

- [Rust Client Quickstart](/quickstart/getting-started-with-the-rust-client) — the lifecycle this page composes against
- [Integrating Cycles with Rust](/how-to/integrating-cycles-with-rust) — broader integration patterns (multi-step, framework middleware)
- [Error Handling in Rust](/how-to/error-handling-patterns-in-rust) — retry, backoff, graceful degradation
- [Rust Client Configuration Reference](/configuration/rust-client-configuration-reference) — full config surface
- [Cost Estimation Cheat Sheet](/how-to/cost-estimation-cheat-sheet) — token-to-dollar mapping across providers
- [How Reserve-Commit Works](/protocol/how-reserve-commit-works-in-cycles) — the underlying lifecycle



# AutoGen Budget Control: Cost Governance for Microsoft AutoGen

This guide shows how to add budget governance to [AutoGen](https://microsoft.github.io/autogen/) multi-agent workflows so that every LLM call is cost-controlled, observable, and automatically stopped when budgets run out.

AutoGen (v0.4+) does not have a built-in middleware or callback system for intercepting LLM calls. The recommended pattern is to wrap the model client with a budget-gated wrapper that creates Cycles reservations before each call and commits actual usage after.

## Prerequisites

```bash
pip install runcycles autogen-agentchat "autogen-ext[openai]"
```

```bash
export CYCLES_BASE_URL="http://localhost:7878"
export CYCLES_API_KEY="your-api-key"   # create via Admin Server — see note below
export CYCLES_TENANT="acme"
export OPENAI_API_KEY="sk-..."
```

> **Need an API key?** Create one via the Admin Server — see [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack#step-3-create-an-api-key) or [API Key Management](/how-to/api-key-management-in-cycles).

::: tip 60-Second Quick Start
```python
import asyncio
from autogen_agentchat.agents import AssistantAgent
from autogen_ext.models.openai import OpenAIChatCompletionClient
from runcycles import AsyncCyclesClient, CyclesConfig, cycles, set_default_client

# `ask` below is async, so the default client must be an AsyncCyclesClient
set_default_client(AsyncCyclesClient(CyclesConfig.from_env()))
model_client = OpenAIChatCompletionClient(model="gpt-4o")

@cycles(estimate=2_000_000, action_kind="llm.completion", action_name="gpt-4o")
async def ask(prompt: str) -> str:
    agent = AssistantAgent("assistant", model_client=model_client)
    result = await agent.run(task=prompt)
    await model_client.close()
    return result.messages[-1].content

print(asyncio.run(ask("What is budget authority?")))
```
Every agent run is now budget-guarded. If the budget is exhausted, `BudgetExceededError` is raised _before_ the agent runs. Read on for per-call budget control with a model client wrapper.
:::

## Budget-gated model client

Wrap `OpenAIChatCompletionClient` to create a Cycles reservation before every LLM call and commit actual token usage after:

```python
import uuid
from autogen_ext.models.openai import OpenAIChatCompletionClient
from autogen_core.models import CreateResult, RequestUsage
from runcycles import (
    AsyncCyclesClient, CyclesConfig, ReservationCreateRequest, CommitRequest,
    ReleaseRequest, Subject, Action, Amount, Unit, CyclesMetrics,
    BudgetExceededError, CyclesProtocolError,
)

PRICE_PER_INPUT_TOKEN = 250      # GPT-4o: $2.50/1M tokens in microcents
PRICE_PER_OUTPUT_TOKEN = 1_000   # GPT-4o: $10/1M tokens in microcents


class CyclesBudgetClient:
    """Wraps an OpenAIChatCompletionClient with Cycles budget governance.

    Delegates all ChatCompletionClient protocol methods to the inner client,
    overriding create() to add reserve → execute → commit lifecycle.
    """

    def __init__(
        self,
        inner: OpenAIChatCompletionClient,
        cycles_client: AsyncCyclesClient,
        tenant: str = "acme",
        workflow: str | None = None,
        agent: str | None = None,
        estimate_amount: int = 2_000_000,
    ):
        self._inner = inner
        self._cycles = cycles_client
        self._subject = Subject(tenant=tenant, workflow=workflow, agent=agent)
        self._estimate_amount = estimate_amount

    async def create(self, messages, **kwargs) -> CreateResult:
        key = str(uuid.uuid4())

        # Reserve budget
        res = await self._cycles.create_reservation(ReservationCreateRequest(
            idempotency_key=key,
            subject=self._subject,
            action=Action(kind="llm.completion", name="gpt-4o"),
            estimate=Amount(unit=Unit.USD_MICROCENTS, amount=self._estimate_amount),
            ttl_ms=60_000,
        ))

        if not res.is_success:
            error = res.get_error_response()
            if error and error.error == "BUDGET_EXCEEDED":
                raise BudgetExceededError(
                    error.message, status=res.status,
                    error_code=error.error, request_id=error.request_id,
                )
            msg = error.message if error else (res.error_message or "Reservation failed")
            raise CyclesProtocolError(
                msg, status=res.status,
                error_code=error.error if error else None,
            )

        rid = res.get_body_attribute("reservation_id")

        try:
            # Execute LLM call
            result = await self._inner.create(messages, **kwargs)

            # Commit actual cost
            input_tokens = result.usage.prompt_tokens if result.usage else 0
            output_tokens = result.usage.completion_tokens if result.usage else 0
            actual = input_tokens * PRICE_PER_INPUT_TOKEN + output_tokens * PRICE_PER_OUTPUT_TOKEN

            await self._cycles.commit_reservation(rid, CommitRequest(
                idempotency_key=f"commit-{key}",
                actual=Amount(unit=Unit.USD_MICROCENTS, amount=actual),
                metrics=CyclesMetrics(
                    tokens_input=input_tokens,
                    tokens_output=output_tokens,
                ),
            ))

            return result
        except BudgetExceededError:
            raise
        except Exception:
            await self._cycles.release_reservation(
                rid, ReleaseRequest(idempotency_key=f"release-{key}"),
            )
            raise

    def create_stream(self, messages, **kwargs):
        # Streaming calls are delegated without budget governance.
        # For per-stream budget control, use client.stream_reservation(...) —
        # see /how-to/handling-streaming-responses-with-cycles.
        return self._inner.create_stream(messages, **kwargs)

    async def close(self):
        await self._inner.close()

    def actual_usage(self):
        return self._inner.actual_usage()

    def total_usage(self):
        return self._inner.total_usage()

    def count_tokens(self, messages, *, tools=[]):
        return self._inner.count_tokens(messages, tools=tools)

    def remaining_tokens(self, messages, *, tools=[]):
        return self._inner.remaining_tokens(messages, tools=tools)

    @property
    def capabilities(self):
        return self._inner.capabilities

    @property
    def model_info(self):
        return self._inner.model_info
```

Streaming calls (`create_stream`) pass through without governance in this wrapper. For per-stream budget control, use `client.stream_reservation(...)` — see [Handling Streaming Responses](/how-to/handling-streaming-responses-with-cycles).

## Using the budget-gated client

### Single agent

Pass the wrapped client to any `AssistantAgent`:

```python
import asyncio
from autogen_agentchat.agents import AssistantAgent
from autogen_ext.models.openai import OpenAIChatCompletionClient
from runcycles import AsyncCyclesClient, CyclesConfig, BudgetExceededError

cycles_client = AsyncCyclesClient(CyclesConfig.from_env())
inner = OpenAIChatCompletionClient(model="gpt-4o")

model = CyclesBudgetClient(
    inner=inner,
    cycles_client=cycles_client,
    tenant="acme",
    agent="support-bot",
)

agent = AssistantAgent("support-bot", model_client=model)

async def main():
    try:
        result = await agent.run(task="Explain budget governance for AI agents.")
        print(result.messages[-1].content)
    except BudgetExceededError:
        print("Budget exhausted.")
    finally:
        await model.close()

asyncio.run(main())
```

### With tools

Every LLM call the agent makes — including tool-calling turns — gets its own reservation:

```python
from autogen_core.tools import FunctionTool
from autogen_agentchat.agents import AssistantAgent

async def get_weather(location: str) -> str:
    """Get current weather for a location."""
    return f"72°F and sunny in {location}"

weather_tool = FunctionTool(get_weather, description="Get current weather")

agent = AssistantAgent(
    "weather-agent",
    model_client=model,
    tools=[weather_tool],
    system_message="Use the weather tool to answer questions.",
)

result = await agent.run(task="What's the weather in NYC?")
```

Each iteration of the tool-calling loop (LLM call → tool → LLM call) creates its own reservation. The agent stops as soon as budget is denied.

## Per-agent budget scoping in teams

Use separate `CyclesBudgetClient` instances with different `agent` values for each team member:

```python
from autogen_agentchat.teams import RoundRobinGroupChat
from autogen_agentchat.conditions import MaxMessageTermination

inner = OpenAIChatCompletionClient(model="gpt-4o")

researcher_model = CyclesBudgetClient(
    inner=inner,
    cycles_client=cycles_client,
    tenant="acme",
    workflow="research-pipeline",
    agent="researcher",
    estimate_amount=3_000_000,
)

writer_model = CyclesBudgetClient(
    inner=inner,
    cycles_client=cycles_client,
    tenant="acme",
    workflow="research-pipeline",
    agent="writer",
    estimate_amount=2_000_000,
)

researcher = AssistantAgent(
    "researcher",
    model_client=researcher_model,
    system_message="Research topics thoroughly and provide detailed findings.",
)

writer = AssistantAgent(
    "writer",
    model_client=writer_model,
    system_message="Write clear, concise reports from research findings.",
)

team = RoundRobinGroupChat(
    participants=[researcher, writer],
    termination_condition=MaxMessageTermination(max_messages=6),
)

try:
    result = await team.run(task="Analyze AI safety trends for Q4.")
    print(result.messages[-1].content)
except BudgetExceededError:
    print("Team stopped — budget exhausted.")
```

This gives you a budget hierarchy: `tenant (acme)` > `workflow (research-pipeline)` > `agent (researcher / writer)`. Each agent can have its own budget limits set by the budget authority.

## Guarding entire workflows with the decorator

For coarser-grained control — budgeting the entire team run rather than individual LLM calls — use the `@cycles` decorator:

```python
from runcycles import cycles, set_default_client, BudgetExceededError

# The decorated function is async, so it needs an AsyncCyclesClient
set_default_client(AsyncCyclesClient(CyclesConfig.from_env()))

@cycles(estimate=10_000_000, action_kind="llm.completion", action_name="research-pipeline")
async def run_research_pipeline(topic: str) -> str:
    result = await team.run(task=f"Research and write a report on: {topic}")
    return result.messages[-1].content

try:
    report = await run_research_pipeline("quantum computing")
    print(report)
except BudgetExceededError:
    print("Pipeline budget exhausted.")
```

With this approach, the entire team run gets a single reservation. This is simpler but less granular than per-call wrapping.

## Swarm teams with budget governance

For `Swarm` teams where agents hand off to each other, each agent's model client tracks its own budget:

```python
from autogen_agentchat.teams import Swarm
from autogen_agentchat.conditions import MaxMessageTermination

reviewer = AssistantAgent(
    "reviewer",
    model_client=CyclesBudgetClient(
        inner=inner, cycles_client=cycles_client,
        tenant="acme", agent="reviewer",
    ),
    handoffs=["approver"],
    system_message="Review budgets. Hand off to approver when ready.",
)

approver = AssistantAgent(
    "approver",
    model_client=CyclesBudgetClient(
        inner=inner, cycles_client=cycles_client,
        tenant="acme", agent="approver",
    ),
    handoffs=["reviewer"],
    system_message="Approve or reject. Hand back to reviewer if issues found.",
)

team = Swarm(
    participants=[reviewer, approver],
    termination_condition=MaxMessageTermination(max_messages=10),
)

result = await team.run(task="Review this budget proposal: ...")
```

## Choosing an integration approach

| Approach | Granularity | Best for |
|----------|------------|----------|
| `CyclesBudgetClient` wrapper | Per-LLM-call | Fine-grained token tracking per agent |
| `@cycles` decorator on run | Per-workflow | Coarser budget control, simpler setup |
| Per-agent wrappers in teams | Per-LLM-call, per-agent scoped | Independent budgets per team member |

You can combine approaches — for example, use per-agent `CyclesBudgetClient` wrappers for LLM cost tracking and `@cycles` on the team run for total workflow budget.

## Key points

- **Wrap the model client, not the agent.** AutoGen v0.4+ doesn't have callback hooks, so wrap `OpenAIChatCompletionClient` with `CyclesBudgetClient` for per-call budget governance.
- **Per-agent scoping with separate wrappers.** Create wrappers with different `agent` values to track and limit costs per team member independently.
- **Tool-calling turns are automatically covered.** Each LLM call in a tool-use loop gets its own reservation through the model client wrapper.
- **Everything is async.** AutoGen v0.4+ is fully async — use `AsyncCyclesClient`, and `asyncio.run()` or `await` for all agent and team operations. The `@cycles` decorator requires an `AsyncCyclesClient` when the decorated function is async.
- **Errors stop the agent.** `BudgetExceededError` raised in the model client propagates up and stops the agent or team.

## Next steps

- [Error Handling Patterns in Python](/how-to/error-handling-patterns-in-python) — handling budget errors in Python
- [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) — strategies for graceful degradation
- [Testing with Cycles](/how-to/testing-with-cycles) — testing budget-guarded code
- [Production Operations Guide](/how-to/production-operations-guide) — running Cycles in production
- [Integrating with OpenAI](/how-to/integrating-cycles-with-openai) — budget governance for direct OpenAI calls



# Integrating Cycles with AWS Bedrock

This guide shows how to add budget governance to AWS Bedrock model invocations using the `runcycles` TypeScript client and the `@aws-sdk/client-bedrock-runtime`.

## Prerequisites

- A running Cycles stack with a tenant, API key, and budget ([Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack))
- AWS credentials configured for Bedrock access
- Node.js 20+

## Installation

```bash
npm install runcycles @aws-sdk/client-bedrock-runtime
```

## Non-streaming calls with `withCycles`

For non-streaming `InvokeModel` calls, use the `withCycles` higher-order function. The example intentionally uses Amazon Bedrock's platform-specific Claude Sonnet 4 model ID, which AWS currently classifies as legacy in some regions with an October 14, 2026 end-of-life date. Confirm availability in your region and replace the ID with a supported Bedrock model before deploying.

```typescript
import { InvokeModelCommand, BedrockRuntimeClient } from "@aws-sdk/client-bedrock-runtime";
import { CyclesClient, CyclesConfig, withCycles, setDefaultClient, getCyclesContext } from "runcycles";

const cyclesClient = new CyclesClient(CyclesConfig.fromEnv());
setDefaultClient(cyclesClient);

const bedrock = new BedrockRuntimeClient({ region: "us-east-1" });
const MODEL_ID = "anthropic.claude-sonnet-4-20250514-v1:0";

// Claude on Bedrock pricing (us-east-1):
// Input: $3.00/1M tokens = 300 microcents/token
// Output: $15.00/1M tokens = 1500 microcents/token
function costMicrocents(inputTokens: number, outputTokens: number): number {
  return Math.ceil(inputTokens * 300 + outputTokens * 1500);
}

const DEFAULT_MAX_TOKENS = 1024;

const askClaude = withCycles(
  {
    client: cyclesClient,
    actionKind: "llm.completion",
    actionName: MODEL_ID,
    estimate: (prompt: string) => {
      const inputTokens = Math.ceil(prompt.length / 4);
      return costMicrocents(inputTokens, DEFAULT_MAX_TOKENS);
    },
    actual: (response: { usage: { input_tokens: number; output_tokens: number } }) => {
      return costMicrocents(response.usage.input_tokens, response.usage.output_tokens);
    },
  },
  async (prompt: string) => {
    const ctx = getCyclesContext();

    // Respect budget caps
    let maxTokens = DEFAULT_MAX_TOKENS;
    if (ctx?.caps?.maxTokens) {
      maxTokens = Math.min(maxTokens, ctx.caps.maxTokens);
    }

    const command = new InvokeModelCommand({
      modelId: MODEL_ID,
      contentType: "application/json",
      accept: "application/json",
      body: JSON.stringify({
        anthropic_version: "bedrock-2023-05-31",
        max_tokens: maxTokens,
        messages: [{ role: "user", content: prompt }],
      }),
    });

    const result = await bedrock.send(command);
    const response = JSON.parse(new TextDecoder().decode(result.body));

    // Report metrics for observability
    if (ctx) {
      ctx.metrics = {
        tokensInput: response.usage.input_tokens,
        tokensOutput: response.usage.output_tokens,
        modelVersion: MODEL_ID,
      };
    }

    return response;
  },
);

const response = await askClaude("Explain budget governance for AI agents.");
console.log(response.content[0].text);
```

## Streaming calls with reserveForStream

For streaming responses, use `reserveForStream` to manage the reservation lifecycle manually:

```typescript
import { InvokeModelWithResponseStreamCommand } from "@aws-sdk/client-bedrock-runtime";
import {
  CyclesClient,
  CyclesConfig,
  reserveForStream,
  BudgetExceededError,
} from "runcycles";

const cyclesClient = new CyclesClient(CyclesConfig.fromEnv());
const bedrock = new BedrockRuntimeClient({ region: "us-east-1" });

const MODEL_ID = "anthropic.claude-sonnet-4-20250514-v1:0";
const MAX_TOKENS = 1024;

// Pricing: Claude 3 Sonnet on Bedrock
// Input: $3.00/1M tokens = 300 microcents/token
// Output: $15.00/1M tokens = 1500 microcents/token
function estimateCost(inputTokens: number, maxOutputTokens: number): number {
  return Math.ceil((inputTokens * 300 + maxOutputTokens * 1500) * 1.2);
}

async function streamWithBudget(prompt: string) {
  const estimatedInputTokens = Math.ceil(prompt.length / 4);
  const estimate = estimateCost(estimatedInputTokens, MAX_TOKENS);

  // 1. Reserve budget
  const handle = await reserveForStream({
    client: cyclesClient,
    estimate,
    unit: "USD_MICROCENTS",
    actionKind: "llm.completion",
    actionName: MODEL_ID,
  });

  try {
    // Respect budget caps
    let maxTokens = MAX_TOKENS;
    if (handle.caps?.maxTokens) {
      maxTokens = Math.min(maxTokens, handle.caps.maxTokens);
    }

    // 2. Stream the response
    const command = new InvokeModelWithResponseStreamCommand({
      modelId: MODEL_ID,
      contentType: "application/json",
      accept: "application/json",
      body: JSON.stringify({
        anthropic_version: "bedrock-2023-05-31",
        max_tokens: maxTokens,
        messages: [{ role: "user", content: prompt }],
      }),
    });

    const result = await bedrock.send(command);

    let inputTokens = 0;
    let outputTokens = 0;

    if (result.body) {
      for await (const event of result.body) {
        if (event.chunk?.bytes) {
          const chunk = JSON.parse(new TextDecoder().decode(event.chunk.bytes));

          if (chunk.type === "content_block_delta" && chunk.delta?.text) {
            process.stdout.write(chunk.delta.text);
          }
          if (chunk.type === "message_start" && chunk.message?.usage) {
            inputTokens = chunk.message.usage.input_tokens;
          }
          if (chunk.type === "message_delta" && chunk.usage?.output_tokens) {
            outputTokens = chunk.usage.output_tokens;
          }
        }
      }
    }

    // 3. Commit actual usage
    const actualCost = Math.ceil(inputTokens * 300 + outputTokens * 1500);
    await handle.commit(actualCost, {
      tokensInput: inputTokens,
      tokensOutput: outputTokens,
      modelVersion: MODEL_ID,
    });
  } catch (err) {
    await handle.release("stream_error");
    throw err;
  }
}
```

## Bedrock token usage extraction

Bedrock streams usage metadata in specific event types:

- **`message_start`** — contains `message.usage.input_tokens`
- **`message_delta`** — contains `usage.output_tokens`

Track both to calculate accurate actual cost for the commit.

## Next steps

- [Handling Streaming Responses](/how-to/handling-streaming-responses-with-cycles) — streaming patterns in detail
- [Cost Estimation Cheat Sheet](/how-to/cost-estimation-cheat-sheet) — pricing reference for estimation
- [Error Handling in TypeScript](/how-to/error-handling-patterns-in-typescript) — handling budget errors
- [AWS Bedrock example (TypeScript)](https://github.com/runcycles/cycles-client-typescript/tree/main/examples/aws-bedrock) — runnable AWS Bedrock integration



# CrewAI Budget Control: Multi-Agent Cost Enforcement

This guide shows how to add budget management to CrewAI multi-agent workflows so that every agent task is cost-controlled, observable, and automatically stopped when budgets run out.

## Prerequisites

```bash
pip install runcycles crewai
```

```bash
export CYCLES_BASE_URL="http://localhost:7878"
export CYCLES_API_KEY="your-api-key"   # create via Admin Server — see note below
export CYCLES_TENANT="acme"
export OPENAI_API_KEY="sk-..."
```

> **Need an API key?** Create one via the Admin Server — see [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack#step-3-create-an-api-key) or [API Key Management](/how-to/api-key-management-in-cycles).

::: tip 60-Second Quick Start
```python
from crewai import Agent, Task, Crew
from runcycles import CyclesClient, CyclesConfig, cycles, set_default_client

set_default_client(CyclesClient(CyclesConfig.from_env()))

researcher = Agent(role="Researcher", goal="Find key facts", backstory="Expert researcher")

@cycles(estimate=3_000_000, action_kind="llm.completion", action_name="crew-research")
def run_research(topic: str) -> str:
    task = Task(description=f"Research {topic}", agent=researcher, expected_output="Summary")
    crew = Crew(agents=[researcher], tasks=[task])
    result = crew.kickoff()
    return str(result)

print(run_research("renewable energy trends"))
```
Every crew execution is now budget-guarded. If the budget is exhausted, `BudgetExceededError` is raised _before_ CrewAI runs. Read on for per-agent patterns.
:::

## Guarding individual agent tasks

Wrap each task function separately so you get per-task cost visibility:

```python
from crewai import Agent, Task, Crew
from runcycles import (
    CyclesClient, CyclesConfig, CyclesMetrics,
    cycles, get_cycles_context, set_default_client, BudgetExceededError,
)

config = CyclesConfig.from_env()
set_default_client(CyclesClient(config))

researcher = Agent(role="Researcher", goal="Find key facts", backstory="Expert researcher")
writer = Agent(role="Writer", goal="Write clear reports", backstory="Technical writer")

@cycles(estimate=2_000_000, action_kind="llm.completion", action_name="research-task")
def run_research(topic: str) -> str:
    task = Task(
        description=f"Research the latest developments in {topic}",
        agent=researcher,
        expected_output="Bullet-point summary of key findings",
    )
    crew = Crew(agents=[researcher], tasks=[task])
    return str(crew.kickoff())

@cycles(estimate=2_500_000, action_kind="llm.completion", action_name="writing-task")
def run_writing(research_results: str) -> str:
    task = Task(
        description=f"Write a report based on: {research_results}",
        agent=writer,
        expected_output="A well-structured report",
    )
    crew = Crew(agents=[writer], tasks=[task])
    return str(crew.kickoff())

# Pipeline: research then write, each independently budget-guarded
research = run_research("AI safety")
report = run_writing(research)
```

## Per-agent budget scoping

Use the `agent` parameter on the decorator to scope budgets per agent. This lets the budget authority set different limits for each agent role:

```python
@cycles(
    estimate=2_000_000,
    action_kind="llm.completion",
    action_name="research-task",
    agent="researcher",
)
def run_research(topic: str) -> str:
    task = Task(
        description=f"Research {topic}",
        agent=researcher,
        expected_output="Summary",
    )
    crew = Crew(agents=[researcher], tasks=[task])
    return str(crew.kickoff())

@cycles(
    estimate=2_500_000,
    action_kind="llm.completion",
    action_name="writing-task",
    agent="writer",
)
def run_writing(research_results: str) -> str:
    task = Task(
        description=f"Write a report based on: {research_results}",
        agent=writer,
        expected_output="Report",
    )
    crew = Crew(agents=[writer], tasks=[task])
    return str(crew.kickoff())
```

With this setup, the budget authority can allocate separate budgets for `researcher` and `writer` under the same tenant.

## Multi-crew budget hierarchies

For complex deployments, use the `tenant`, `workspace`, and `agent` parameters to create hierarchical budget scoping across multiple crews:

```python
# Crew 1: Content team
@cycles(
    estimate=2_000_000,
    action_kind="llm.completion",
    action_name="content-research",
    tenant="acme",
    workspace="content-team",
    agent="researcher",
)
def content_research(topic: str) -> str:
    task = Task(description=f"Research {topic}", agent=researcher, expected_output="Summary")
    crew = Crew(agents=[researcher], tasks=[task])
    return str(crew.kickoff())

# Crew 2: Engineering team
@cycles(
    estimate=3_000_000,
    action_kind="llm.completion",
    action_name="code-review",
    tenant="acme",
    workspace="engineering-team",
    agent="code-reviewer",
)
def code_review(code: str) -> str:
    reviewer = Agent(role="Code Reviewer", goal="Review code", backstory="Senior engineer")
    task = Task(description=f"Review this code:\n{code}", agent=reviewer, expected_output="Review")
    crew = Crew(agents=[reviewer], tasks=[task])
    return str(crew.kickoff())
```

This gives you a budget hierarchy: `tenant (acme)` > `workspace (content-team / engineering-team)` > `agent (researcher / code-reviewer)`. Each level can have its own budget limits set by the budget authority.

::: tip Callable subject and action fields (runcycles 0.4.0+)
The subject (`tenant`, `workspace`, `agent`, ...) and action (`action_kind`, `action_name`, ...) parameters on `@cycles` also accept callables that receive the decorated function's arguments at reservation time — so a single decorated function can route budgets per crew or per tenant:

```python
@cycles(
    estimate=2_000_000,
    action_kind="llm.completion",
    action_name=lambda topic, **kw: kw.get("crew_name", "default-crew"),
    tenant=lambda topic, **kw: kw.get("tenant", "acme"),
)
def run_crew_task(topic: str, tenant: str = "acme", crew_name: str = "default-crew") -> str:
    ...
```
:::

## Error handling

When a budget is insufficient, `BudgetExceededError` is raised **before** CrewAI executes:

```python
from runcycles import BudgetExceededError

try:
    research = run_research("quantum computing")
    report = run_writing(research)
except BudgetExceededError:
    report = "Budget limit reached. Deferring this task to the next billing cycle."
```

For multi-step pipelines, handle errors at each stage to allow partial completion:

```python
try:
    research = run_research("quantum computing")
except BudgetExceededError:
    research = cached_research.get("quantum computing", "No data available.")

try:
    report = run_writing(research)
except BudgetExceededError:
    report = f"Raw research (report generation skipped):\n{research}"
```

See [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) for patterns like queueing, model downgrade, and caching.

## Key points

- **Wrap task functions, not agents.** The `@cycles` decorator goes on your functions that invoke CrewAI, giving you budget control at the task level.
- **Use `agent` for per-agent scoping.** The `agent` parameter lets the budget authority allocate and track costs per agent role.
- **Budget hierarchies map to org structure.** Use `tenant`, `workspace`, and `agent` to mirror your team and crew topology.
- **The function never executes on DENY.** CrewAI agents never run if the budget is exhausted, saving both cost and compute.

## Next steps

- [Error Handling Patterns in Python](/how-to/error-handling-patterns-in-python) — handling budget errors in Python
- [Testing with Cycles](/how-to/testing-with-cycles) — testing budget-guarded code
- [Integrating with OpenAI](/how-to/integrating-cycles-with-openai) — budget governance for direct OpenAI calls
- [Integrating with LangChain](/how-to/integrating-cycles-with-langchain) — budget governance for LangChain apps
- [Production Operations Guide](/how-to/production-operations-guide) — running Cycles in production



# Integrating Cycles with Django

This guide shows how to add budget management to a Django application using middleware, per-tenant isolation, and exception handling.

## Prerequisites

```bash
pip install runcycles django
```

```bash
export CYCLES_BASE_URL="http://localhost:7878"
export CYCLES_API_KEY="your-api-key"   # create via Admin Server — see note below
export CYCLES_TENANT="acme"
```

> **Need an API key?** Create one via the Admin Server — see [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack#step-3-create-an-api-key) or [API Key Management](/how-to/api-key-management-in-cycles).

## Client initialization

Create a Cycles client that lives for the process lifetime. Use Django's `AppConfig.ready()` hook:

```python
# myapp/apps.py
from django.apps import AppConfig
from runcycles import CyclesClient, CyclesConfig, set_default_client

class MyAppConfig(AppConfig):
    name = "myapp"

    def ready(self):
        client = CyclesClient(CyclesConfig.from_env())
        set_default_client(client)
        # Store on the module for direct access
        import myapp
        myapp.cycles_client = client
```

Setting the default client means `@cycles`-decorated functions work without passing `client=` explicitly.

## Preflight middleware

Use `client.decide()` to check budget before processing expensive requests:

```python
# myapp/middleware.py
import uuid
from django.http import JsonResponse
from runcycles import DecisionRequest, Subject, Action, Amount, Unit

BUDGET_GUARDED_PATHS = {"/api/chat/", "/api/summarize/"}

class CyclesBudgetMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        if request.path not in BUDGET_GUARDED_PATHS:
            return self.get_response(request)

        import myapp
        client = myapp.cycles_client
        tenant = request.headers.get("X-Tenant-ID", "acme")

        response = client.decide(DecisionRequest(
            idempotency_key=str(uuid.uuid4()),
            subject=Subject(tenant=tenant, app="my-django-api"),
            action=Action(kind="api.request", name=request.path),
            estimate=Amount(unit=Unit.USD_MICROCENTS, amount=1_000_000),
        ))

        if response.is_success:
            decision = response.get_body_attribute("decision")
            if decision == "DENY":
                return JsonResponse(
                    {"error": "budget_exceeded", "message": "Insufficient budget."},
                    status=402,
                )

        return self.get_response(request)
```

Add the middleware to `settings.py`:

```python
# settings.py
MIDDLEWARE = [
    # ... existing middleware ...
    "myapp.middleware.CyclesBudgetMiddleware",
]
```

## Exception handling middleware

Convert Cycles exceptions into appropriate HTTP responses:

```python
# myapp/middleware.py
from django.http import JsonResponse
from runcycles import BudgetExceededError, CyclesProtocolError

class CyclesExceptionMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        return self.get_response(request)

    def process_exception(self, request, exception):
        if isinstance(exception, BudgetExceededError):
            return JsonResponse(
                {
                    "error": "budget_exceeded",
                    "message": "Insufficient budget for this request.",
                    "retry_after_ms": exception.retry_after_ms,
                },
                status=402,
            )
        if isinstance(exception, CyclesProtocolError):
            status = 429 if exception.is_retryable() else 503
            return JsonResponse(
                {
                    "error": str(exception.error_code),
                    "message": str(exception),
                    "retry_after_ms": exception.retry_after_ms,
                },
                status=status,
            )
        return None
```

Add it to `MIDDLEWARE` (before `CyclesBudgetMiddleware`):

```python
MIDDLEWARE = [
    # ... existing middleware ...
    "myapp.middleware.CyclesExceptionMiddleware",
    "myapp.middleware.CyclesBudgetMiddleware",
]
```

## Budget-guarded views

Use the `@cycles` decorator on view functions or helper functions:

```python
# myapp/views.py
import json
from django.http import JsonResponse
from django.views.decorators.http import require_POST
from runcycles import cycles, get_cycles_context, CyclesMetrics

PRICE_PER_INPUT_TOKEN = 250
PRICE_PER_OUTPUT_TOKEN = 1_000

@cycles(
    estimate=lambda prompt, **kw: len(prompt.split()) * 2 * PRICE_PER_INPUT_TOKEN
        + kw.get("max_tokens", 1024) * PRICE_PER_OUTPUT_TOKEN,
    actual=lambda result: result["cost"],
    action_kind="llm.completion",
    action_name="gpt-4o",
    unit="USD_MICROCENTS",
)
def guarded_llm_call(prompt: str, max_tokens: int = 1024) -> dict:
    ctx = get_cycles_context()
    if ctx and ctx.has_caps() and ctx.caps.max_tokens:
        max_tokens = min(max_tokens, ctx.caps.max_tokens)

    # Your LLM call here
    response = call_llm(prompt, max_tokens=max_tokens)

    if ctx:
        ctx.metrics = CyclesMetrics(
            tokens_input=response["usage"]["input_tokens"],
            tokens_output=response["usage"]["output_tokens"],
        )

    return {
        "content": response["content"],
        "cost": (response["usage"]["input_tokens"] * PRICE_PER_INPUT_TOKEN
                 + response["usage"]["output_tokens"] * PRICE_PER_OUTPUT_TOKEN),
    }

@require_POST
def chat_view(request):
    body = json.loads(request.body)
    result = guarded_llm_call(body["prompt"])
    return JsonResponse({"response": result["content"]})
```

## Per-tenant isolation

Extract the tenant from request headers and scope budgets per tenant. Subject fields on `@cycles` accept callables (runcycles 0.4.0+) that are invoked with the decorated function's arguments at reservation time — so the tenant can be resolved per call:

```python
# myapp/views.py
from runcycles import cycles

def get_tenant(request) -> str:
    return request.headers.get("X-Tenant-ID", "acme")

@cycles(
    estimate=1_000_000,
    action_kind="llm.completion",
    action_name="gpt-4o",
    # Resolved from this call's kwargs; returning None falls back to the
    # client-config default (CYCLES_TENANT)
    tenant=lambda prompt, **kw: kw.get("tenant", "acme"),
)
def tenant_scoped_call(prompt: str, tenant: str = "acme") -> dict:
    ...

@require_POST
def chat_view(request):
    body = json.loads(request.body)
    tenant = get_tenant(request)
    result = tenant_scoped_call(body["prompt"], tenant=tenant)
    return JsonResponse({"response": result["content"]})
```

## Budget dashboard endpoint

Expose per-tenant budget information:

```python
# myapp/views.py
from django.http import JsonResponse

def budget_view(request, tenant_id):
    import myapp
    client = myapp.cycles_client
    response = client.get_balances(tenant=tenant_id)
    if not response.is_success:
        return JsonResponse({"error": response.error_message}, status=500)
    return JsonResponse(response.body)
```

```python
# urls.py
from django.urls import path
from myapp import views

urlpatterns = [
    path("api/chat/", views.chat_view),
    path("api/budget/<str:tenant_id>/", views.budget_view),
]
```

## Key points

- **Use `CyclesClient` (sync)** in Django — Django views are synchronous by default. Use `AsyncCyclesClient` only with async views.
- **Initialize in `AppConfig.ready()`** — create the client once at startup.
- **Map HTTP errors** — `BudgetExceededError` → 402, retryable errors → 429.
- **Preflight with `decide()`** — lightweight budget check before expensive work.
- **Isolate tenants** — use the `Subject.tenant` field from request headers.
- **Set a default client** — avoids passing `client=` to every `@cycles` decorator.

## Next steps

- [Integrating with FastAPI](/how-to/integrating-cycles-with-fastapi) — async Python web framework integration
- [Error Handling Patterns in Python](/how-to/error-handling-patterns-in-python) — handling budget errors in Python
- [Testing with Cycles](/how-to/testing-with-cycles) — testing budget-guarded code
- [Production Operations Guide](/how-to/production-operations-guide) — running Cycles in production



# Integrating Cycles with Express

This guide shows how to add budget governance to an Express.js application using reusable middleware.

## Prerequisites

- A running Cycles stack with a tenant, API key, and budget ([Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack))
- Node.js 20+

## Installation

```bash
npm install runcycles express
```

## Pattern overview

Two patterns work well with Express:

1. **Middleware pattern** — for routes where every request needs budget governance (e.g., chat endpoints). The middleware reserves budget and attaches a handle to `res.locals`.
2. **Inline pattern** — for routes where budget governance is conditional or has custom logic. Use `withCycles` directly in the route handler.

## Middleware pattern

Create a reusable middleware that reserves budget for each request:

```typescript
// middleware/cycles-guard.ts
import type { Request, Response, NextFunction } from "express";
import {
  CyclesClient,
  reserveForStream,
  BudgetExceededError,
} from "runcycles";

interface CyclesGuardOptions {
  client: CyclesClient;
  actionKind: string;
  actionName: string;
  estimateFn: (req: Request) => number;
  unit?: string;
  tenantFn?: (req: Request) => string;
}

export function cyclesGuard(options: CyclesGuardOptions) {
  return async (req: Request, res: Response, next: NextFunction) => {
    const estimate = options.estimateFn(req);

    try {
      const handle = await reserveForStream({
        client: options.client,
        estimate,
        unit: options.unit ?? "USD_MICROCENTS",
        actionKind: options.actionKind,
        actionName: options.actionName,
        ...(options.tenantFn && { tenant: options.tenantFn(req) }),
      });

      // Attach the handle so route handlers can commit/release
      res.locals.cyclesHandle = handle;

      // Release budget if the client disconnects.
      // handle.finalized is true once commit() or release() has run,
      // so no manual bookkeeping flag is needed.
      res.on("close", async () => {
        if (!handle.finalized) {
          await handle.release("client_disconnect");
        }
      });

      next();
    } catch (err) {
      if (err instanceof BudgetExceededError) {
        res.status(402).json({
          error: "budget_exceeded",
          message: "Budget exhausted for this operation.",
        });
        return;
      }
      next(err);
    }
  };
}
```

Use the middleware on a route:

```typescript
// server.ts
import express from "express";
import { CyclesClient, CyclesConfig } from "runcycles";
import { cyclesGuard } from "./middleware/cycles-guard.js";

const app = express();
app.use(express.json());

const cyclesClient = new CyclesClient(CyclesConfig.fromEnv());

// Protect the chat route with budget governance
app.post(
  "/api/chat",
  cyclesGuard({
    client: cyclesClient,
    actionKind: "llm.completion",
    actionName: "gpt-4o",
    estimateFn: (req) => {
      const messages = req.body?.messages ?? [];
      const chars = messages.reduce(
        (sum: number, m: { content?: string }) =>
          sum + (typeof m.content === "string" ? m.content.length : 0),
        0,
      );
      const inputTokens = Math.ceil(chars / 4);
      return Math.ceil(inputTokens * 250 + inputTokens * 2 * 1000);
    },
  }),
  async (req, res) => {
    const handle = res.locals.cyclesHandle;

    try {
      // Your LLM call here
      const response = await callOpenAI(req.body.messages);

      // Commit actual cost
      const actualCost = calculateActualCost(response.usage);
      await handle.commit(actualCost, {
        tokensInput: response.usage.prompt_tokens,
        tokensOutput: response.usage.completion_tokens,
      });

      res.json({ message: response.content });
    } catch (err) {
      await handle.release("handler_error");
      throw err;
    }
  },
);

app.listen(3000);
```

Note: `throw err` inside an async route handler only reaches Express's error handling on Express 5 — on Express 4, call `next(err)` instead.

## Inline pattern with withCycles

For simpler routes, use `withCycles` directly:

```typescript
import {
  withCycles, CyclesClient, CyclesConfig, setDefaultClient, BudgetExceededError,
} from "runcycles";

const cyclesClient = new CyclesClient(CyclesConfig.fromEnv());
setDefaultClient(cyclesClient);

const summarize = withCycles(
  { estimate: 3000000, actionKind: "llm.completion", actionName: "gpt-4o-mini" },
  async (text: string) => {
    return await callOpenAI([{ role: "user", content: `Summarize: ${text}` }]);
  },
);

app.post("/api/summarize", async (req, res) => {
  try {
    const result = await summarize(req.body.text);
    res.json({ summary: result });
  } catch (err) {
    if (err instanceof BudgetExceededError) {
      res.status(402).json({ error: "budget_exceeded" });
      return;
    }
    throw err;
  }
});
```

## Budget observability endpoint

Add an endpoint to check current budget status:

```typescript
app.get("/api/balance", async (_req, res) => {
  const balances = await cyclesClient.getBalances({
    tenant: cyclesClient.config.tenant!,
  });
  res.json(balances.body);
});
```

## Per-tenant middleware

For multi-tenant applications, resolve the tenant from the request:

```typescript
app.post(
  "/api/chat",
  cyclesGuard({
    client: cyclesClient,
    actionKind: "llm.completion",
    actionName: "gpt-4o",
    estimateFn: (req) => Math.ceil(req.body.text.length / 4 * 1250),
    // Tenant resolved per-request from auth middleware
    tenantFn: (req) => req.auth.tenantId,
  }),
  chatHandler,
);
```

## Streaming responses

For SSE or streaming endpoints, use the programmatic `CyclesClient` with `reserveForStream` instead of the middleware pattern. The middleware commits when the response finishes, but streaming requires manual commit after the stream completes. See [Handling Streaming Responses](/how-to/handling-streaming-responses-with-cycles) for the full pattern.

## Next steps

- [Handling Streaming Responses](/how-to/handling-streaming-responses-with-cycles) — budget-managed streaming with `reserveForStream`
- [Choosing the Right Integration Pattern](/how-to/choosing-the-right-integration-pattern) — when to use middleware vs inline
- [Cost Estimation Cheat Sheet](/how-to/cost-estimation-cheat-sheet) — how much to reserve per model
- [Error Handling in TypeScript](/how-to/error-handling-patterns-in-typescript) — handling Cycles errors
- [Express middleware example](https://github.com/runcycles/cycles-client-typescript/tree/main/examples/express-middleware) — runnable Express middleware integration



# Integrating Cycles with FastAPI

This guide shows how to add budget management to a FastAPI application using middleware, dependency injection, per-tenant isolation, and exception handling.

## Prerequisites

```bash
pip install runcycles fastapi uvicorn
```

```bash
export CYCLES_BASE_URL="http://localhost:7878"
export CYCLES_API_KEY="your-api-key"   # create via Admin Server — see note below
export CYCLES_TENANT="acme"
```

> **Need an API key?** Create one via the Admin Server — see [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack#step-3-create-an-api-key) or [API Key Management](/how-to/api-key-management-in-cycles).

## Client lifecycle

Use FastAPI's lifespan to manage the `AsyncCyclesClient`:

```python
from contextlib import asynccontextmanager
from fastapi import FastAPI
from runcycles import AsyncCyclesClient, CyclesConfig, set_default_client

@asynccontextmanager
async def lifespan(app: FastAPI):
    client = AsyncCyclesClient(CyclesConfig.from_env())
    set_default_client(client)
    app.state.cycles_client = client
    yield
    await client.aclose()

app = FastAPI(lifespan=lifespan)
```

Setting the default client means `@cycles`-decorated functions work without passing `client=` explicitly.

## Exception handlers

Convert Cycles exceptions into appropriate HTTP responses:

```python
from fastapi import Request
from fastapi.responses import JSONResponse
from runcycles import BudgetExceededError, CyclesProtocolError

@app.exception_handler(BudgetExceededError)
async def budget_exceeded_handler(request: Request, exc: BudgetExceededError):
    return JSONResponse(
        status_code=402,
        content={
            "error": "budget_exceeded",
            "message": "Insufficient budget for this request.",
            "retry_after_ms": exc.retry_after_ms,
        },
    )

@app.exception_handler(CyclesProtocolError)
async def protocol_error_handler(request: Request, exc: CyclesProtocolError):
    status = 429 if exc.is_retryable() else 503
    return JSONResponse(
        status_code=status,
        content={
            "error": str(exc.error_code),
            "message": str(exc),
            "retry_after_ms": exc.retry_after_ms,
        },
    )
```

## Preflight middleware

Use `client.decide()` to check budget before processing expensive requests. This avoids starting work that will be denied:

```python
import uuid
from runcycles import DecisionRequest, Subject, Action, Amount, Unit

@app.middleware("http")
async def budget_preflight(request: Request, call_next):
    if request.url.path not in ("/chat", "/summarize"):
        return await call_next(request)

    tenant = request.headers.get("X-Tenant-ID", "acme")
    client = request.app.state.cycles_client

    response = await client.decide(DecisionRequest(
        idempotency_key=str(uuid.uuid4()),
        subject=Subject(tenant=tenant, app="my-api"),
        action=Action(kind="api.request", name=request.url.path),
        estimate=Amount(unit=Unit.USD_MICROCENTS, amount=1_000_000),
    ))

    if response.is_success:
        decision = response.get_body_attribute("decision")
        if decision == "DENY":
            return JSONResponse(
                status_code=402,
                content={"error": "budget_exceeded"},
            )

    return await call_next(request)
```

## Per-tenant isolation

Use Cycles' subject hierarchy to isolate budgets per tenant. Extract the tenant from request headers, and pass it through to a subject callable on the decorator — subject fields on `@cycles` accept callables (runcycles 0.4.0+) that are invoked with the decorated function's arguments at reservation time:

```python
from fastapi import Header, Depends
from runcycles import cycles, get_cycles_context, CyclesMetrics

def get_tenant(x_tenant_id: str = Header(default="acme")) -> str:
    return x_tenant_id

@cycles(
    estimate=lambda prompt, **kw: kw.get("max_tokens", 256) * 1_000,
    actual=lambda result: result.get("cost", 0),
    action_kind="llm.completion",
    action_name="gpt-4o",
    unit="USD_MICROCENTS",
    # Resolved from this call's kwargs; returning None falls back to the
    # client-config default (CYCLES_TENANT)
    tenant=lambda prompt, **kw: kw.get("tenant", "acme"),
)
async def guarded_llm_call(prompt: str, tenant: str = "acme") -> dict:
    ...

@app.get("/chat")
async def chat(prompt: str, tenant: str = Depends(get_tenant)):
    result = await guarded_llm_call(prompt, tenant=tenant)
    return {"response": result["content"]}
```

Each tenant's requests are charged against their own budget scope. If you need control beyond what the decorator offers (custom reservation lifecycles, streaming), drop down to the programmatic client — see [Handling Streaming Responses](/how-to/handling-streaming-responses-with-cycles).

## Budget dashboard endpoint

Expose per-tenant budget information:

```python
from fastapi import HTTPException

@app.get("/budget/{tenant_id}")
async def get_budget(tenant_id: str, request: Request):
    client = request.app.state.cycles_client
    response = await client.get_balances(tenant=tenant_id)
    if not response.is_success:
        raise HTTPException(status_code=500, detail=response.error_message)
    return response.body
```

## Key points

- **Use `AsyncCyclesClient`** in FastAPI — it shares the same async event loop.
- **Manage lifecycle with lifespan** — create the client on startup, close on shutdown.
- **Map HTTP errors** — `BudgetExceededError` → 402, retryable errors → 429.
- **Preflight with `decide()`** — lightweight budget check before expensive work.
- **Isolate tenants** — use the `Subject.tenant` field from request headers.
- **Set a default client** — avoids passing `client=` to every `@cycles` decorator.

## Full example

See [`examples/fastapi_integration.py`](https://github.com/runcycles/cycles-client-python/blob/main/examples/fastapi_integration.py) for a complete, runnable server.

## Next steps

- [Error Handling Patterns in Python](/how-to/error-handling-patterns-in-python) — handling budget errors in Python
- [Handling Streaming Responses](/how-to/handling-streaming-responses-with-cycles) — budget-managed streaming
- [Testing with Cycles](/how-to/testing-with-cycles) — testing budget-guarded code
- [Production Operations Guide](/how-to/production-operations-guide) — running Cycles in production
- [FastAPI example (Python)](https://github.com/runcycles/cycles-client-python/blob/main/examples/fastapi_integration.py) — runnable FastAPI integration



# Integrating Cycles with Flask

This guide shows how to add budget management to a Flask application using error handlers, per-tenant isolation, and preflight budget checks.

## Prerequisites

```bash
pip install runcycles flask
```

```bash
export CYCLES_BASE_URL="http://localhost:7878"
export CYCLES_API_KEY="your-api-key"   # create via Admin Server — see note below
export CYCLES_TENANT="acme"
```

> **Need an API key?** Create one via the Admin Server — see [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack#step-3-create-an-api-key) or [API Key Management](/how-to/api-key-management-in-cycles).

## Client initialization

Create a Cycles client at app startup:

```python
from flask import Flask
from runcycles import CyclesClient, CyclesConfig, set_default_client

app = Flask(__name__)

client = CyclesClient(CyclesConfig.from_env())
set_default_client(client)
app.config["CYCLES_CLIENT"] = client
```

Setting the default client means `@cycles`-decorated functions work without passing `client=` explicitly.

## Error handlers

Convert Cycles exceptions into appropriate HTTP responses:

```python
from flask import jsonify
from runcycles import BudgetExceededError, CyclesProtocolError

@app.errorhandler(BudgetExceededError)
def handle_budget_exceeded(exc):
    return jsonify({
        "error": "budget_exceeded",
        "message": "Insufficient budget for this request.",
        "retry_after_ms": exc.retry_after_ms,
    }), 402

@app.errorhandler(CyclesProtocolError)
def handle_protocol_error(exc):
    status = 429 if exc.is_retryable() else 503
    return jsonify({
        "error": str(exc.error_code),
        "message": str(exc),
        "retry_after_ms": exc.retry_after_ms,
    }), status
```

## Budget-guarded routes

Use the `@cycles` decorator on route handler functions or helper functions:

```python
from flask import request, jsonify
from runcycles import cycles, get_cycles_context, CyclesMetrics

PRICE_PER_INPUT_TOKEN = 250
PRICE_PER_OUTPUT_TOKEN = 1_000

@cycles(
    estimate=lambda prompt, **kw: len(prompt.split()) * 2 * PRICE_PER_INPUT_TOKEN
        + kw.get("max_tokens", 1024) * PRICE_PER_OUTPUT_TOKEN,
    actual=lambda result: result["cost"],
    action_kind="llm.completion",
    action_name="gpt-4o",
    unit="USD_MICROCENTS",
    # Subject fields accept callables (runcycles 0.4.0+): resolved from the
    # call's arguments before the reservation is created
    tenant=lambda prompt, **kw: kw.get("tenant", "acme"),
)
def guarded_llm_call(prompt: str, max_tokens: int = 1024, tenant: str = "acme") -> dict:
    ctx = get_cycles_context()
    if ctx and ctx.has_caps() and ctx.caps.max_tokens:
        max_tokens = min(max_tokens, ctx.caps.max_tokens)

    # Your LLM call here
    response = call_llm(prompt, max_tokens=max_tokens)

    if ctx:
        ctx.metrics = CyclesMetrics(
            tokens_input=response["usage"]["input_tokens"],
            tokens_output=response["usage"]["output_tokens"],
        )

    return {
        "content": response["content"],
        "cost": (response["usage"]["input_tokens"] * PRICE_PER_INPUT_TOKEN
                 + response["usage"]["output_tokens"] * PRICE_PER_OUTPUT_TOKEN),
    }

@app.route("/chat", methods=["POST"])
def chat():
    body = request.get_json()
    result = guarded_llm_call(body["prompt"])
    return jsonify({"response": result["content"]})
```

## Preflight budget check

Use `client.decide()` with a `before_request` hook to check budget before processing expensive requests:

```python
import uuid
from flask import request, jsonify, g
from runcycles import DecisionRequest, Subject, Action, Amount, Unit

BUDGET_GUARDED_PATHS = {"/chat", "/summarize"}

@app.before_request
def budget_preflight():
    if request.path not in BUDGET_GUARDED_PATHS:
        return None

    tenant = request.headers.get("X-Tenant-ID", "acme")
    g.tenant = tenant

    response = client.decide(DecisionRequest(
        idempotency_key=str(uuid.uuid4()),
        subject=Subject(tenant=tenant, app="my-flask-api"),
        action=Action(kind="api.request", name=request.path),
        estimate=Amount(unit=Unit.USD_MICROCENTS, amount=1_000_000),
    ))

    if response.is_success:
        decision = response.get_body_attribute("decision")
        if decision == "DENY":
            return jsonify({"error": "budget_exceeded"}), 402

    return None
```

## Per-tenant isolation

Extract the tenant from request headers and scope budgets per tenant. The `tenant=lambda ...` callable on the `guarded_llm_call` decorator above (runcycles 0.4.0+) reads the `tenant` keyword argument on each call and scopes the reservation's subject to that tenant:

```python
from flask import request, g

@app.before_request
def extract_tenant():
    g.tenant = request.headers.get("X-Tenant-ID", "acme")

@app.route("/chat", methods=["POST"])
def chat():
    body = request.get_json()
    result = guarded_llm_call(body["prompt"], tenant=g.tenant)
    return jsonify({"response": result["content"]})
```

Each tenant's requests are charged against their own budget scope. Subject callables that return `None` fall back to the client-config default (`CYCLES_TENANT`).

## Budget dashboard endpoint

Expose per-tenant budget information:

```python
@app.route("/budget/<tenant_id>")
def get_budget(tenant_id):
    response = client.get_balances(tenant=tenant_id)
    if not response.is_success:
        return jsonify({"error": response.error_message}), 500
    return jsonify(response.body)
```

## Key points

- **Use `CyclesClient` (sync)** in Flask — Flask views are synchronous.
- **Initialize at app startup** — create the client once, store in `app.config`.
- **Map HTTP errors** — `BudgetExceededError` → 402, retryable errors → 429.
- **Preflight with `before_request`** — lightweight budget check before expensive work.
- **Isolate tenants** — use `g.tenant` from request headers.
- **Set a default client** — avoids passing `client=` to every `@cycles` decorator.

## Next steps

- [Integrating with Django](/how-to/integrating-cycles-with-django) — Django web framework integration
- [Integrating with FastAPI](/how-to/integrating-cycles-with-fastapi) — async Python web framework integration
- [Error Handling Patterns in Python](/how-to/error-handling-patterns-in-python) — handling budget errors in Python
- [Testing with Cycles](/how-to/testing-with-cycles) — testing budget-guarded code
- [Production Operations Guide](/how-to/production-operations-guide) — running Cycles in production



# Integrating Cycles with Google Gemini

This guide shows how to add budget governance to Google Gemini API calls using the `runcycles` TypeScript client.

::: warning SDK migration
The examples below use `@google/generative-ai`, which Google is replacing with `@google/genai`. The API patterns are similar — see [Google's migration guide](https://ai.google.dev/gemini-api/docs/migrate) for the new SDK. The runcycles integration works the same way with either SDK.
:::

## Prerequisites

- A running Cycles stack with a tenant, API key, and budget ([Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack))
- A Google AI API key (`GOOGLE_API_KEY`)
- Node.js 20+

## Installation

```bash
npm install runcycles @google/generative-ai
```

## Non-streaming calls with withCycles

```typescript
import { GoogleGenerativeAI } from "@google/generative-ai";
import type { GenerateContentResult } from "@google/generative-ai";
import {
  CyclesClient, CyclesConfig, withCycles,
  getCyclesContext, BudgetExceededError,
} from "runcycles";

const cyclesClient = new CyclesClient(CyclesConfig.fromEnv());
const genAI = new GoogleGenerativeAI(process.env.GOOGLE_API_KEY!);

const MODEL = "gemini-2.0-flash";
const MAX_TOKENS = 1024;

// Per-token pricing in USD microcents (prompts ≤ 128k tokens)
// Input: $0.10/1M tokens = 10 microcents/token
// Output: $0.40/1M tokens = 40 microcents/token
function costMicrocents(inputTokens: number, outputTokens: number): number {
  return Math.ceil(inputTokens * 10 + outputTokens * 40);
}

const callGemini = withCycles(
  {
    client: cyclesClient,
    actionKind: "llm.completion",
    actionName: MODEL,
    estimate: (prompt: string) => {
      const inputTokens = Math.ceil(prompt.length / 4);
      return costMicrocents(inputTokens, MAX_TOKENS);
    },
    actual: (result: GenerateContentResult) => {
      const usage = result.response.usageMetadata;
      return costMicrocents(
        usage?.promptTokenCount ?? 0,
        usage?.candidatesTokenCount ?? 0,
      );
    },
  },
  async (prompt: string) => {
    const ctx = getCyclesContext();

    // Respect budget caps — reduce max_tokens if budget is running low
    let maxTokens = MAX_TOKENS;
    if (ctx?.caps?.maxTokens) {
      maxTokens = Math.min(maxTokens, ctx.caps.maxTokens);
    }

    const model = genAI.getGenerativeModel({
      model: MODEL,
      generationConfig: { maxOutputTokens: maxTokens },
    });

    const result = await model.generateContent(prompt);

    // Report metrics for observability
    if (ctx) {
      const usage = result.response.usageMetadata;
      ctx.metrics = {
        tokensInput: usage?.promptTokenCount,
        tokensOutput: usage?.candidatesTokenCount,
        modelVersion: MODEL,
      };
    }

    return result;
  },
);

// Usage
try {
  const result = await callGemini("Explain budget governance for AI agents.");
  console.log(result.response.text());
} catch (err) {
  if (err instanceof BudgetExceededError) {
    console.error("Budget exhausted:", err.message);
  } else {
    throw err;
  }
}
```

::: tip estimate vs actual
The `estimate` callback runs **before** the LLM call to reserve budget. The `actual` callback runs **after** to commit real usage. Without `actual`, Cycles commits the estimate — which overstates cost on short responses and understates it on long ones. Always provide both when token counts are available.
:::

## Streaming calls with reserveForStream

For streaming responses, use `reserveForStream` to reserve before the stream starts and commit real token counts after it finishes:

```typescript
import { GoogleGenerativeAI } from "@google/generative-ai";
import { CyclesClient, CyclesConfig, reserveForStream, BudgetExceededError } from "runcycles";

const cyclesClient = new CyclesClient(CyclesConfig.fromEnv());
const genAI = new GoogleGenerativeAI(process.env.GOOGLE_API_KEY!);

const MODEL = "gemini-2.0-flash";
const MAX_TOKENS = 1024;

function costMicrocents(inputTokens: number, outputTokens: number): number {
  return Math.ceil(inputTokens * 10 + outputTokens * 40);
}

async function streamWithBudget(prompt: string) {
  const estimatedInputTokens = Math.ceil(prompt.length / 4);
  const estimate = costMicrocents(estimatedInputTokens, MAX_TOKENS);

  // 1. Reserve budget
  const handle = await reserveForStream({
    client: cyclesClient,
    estimate,
    unit: "USD_MICROCENTS",
    actionKind: "llm.completion",
    actionName: MODEL,
  });

  try {
    // Respect budget caps
    let maxTokens = MAX_TOKENS;
    if (handle.caps?.maxTokens) {
      maxTokens = Math.min(maxTokens, handle.caps.maxTokens);
    }

    // 2. Stream the response
    const model = genAI.getGenerativeModel({
      model: MODEL,
      generationConfig: { maxOutputTokens: maxTokens },
    });

    const streamResult = await model.generateContentStream(prompt);

    for await (const chunk of streamResult.stream) {
      const text = chunk.text();
      if (text) process.stdout.write(text);
    }
    console.log();

    // 3. Commit actual usage from aggregated response
    const aggregated = await streamResult.response;
    const usage = aggregated.usageMetadata;
    const inputTokens = usage?.promptTokenCount ?? 0;
    const outputTokens = usage?.candidatesTokenCount ?? 0;

    await handle.commit(costMicrocents(inputTokens, outputTokens), {
      tokensInput: inputTokens,
      tokensOutput: outputTokens,
      modelVersion: MODEL,
    });
  } catch (err) {
    await handle.release("stream_error");
    throw err;
  }
}
```

## Gemini usage metadata

The Gemini SDK provides token counts through `response.usageMetadata`:

- `promptTokenCount` — input tokens
- `candidatesTokenCount` — output tokens
- `totalTokenCount` — total tokens

For streaming, access this from the aggregated response after the stream completes: `const aggregated = await streamResult.response`.

## Next steps

- [Handling Streaming Responses](/how-to/handling-streaming-responses-with-cycles) — streaming patterns in detail
- [Cost Estimation Cheat Sheet](/how-to/cost-estimation-cheat-sheet) — pricing reference for estimation
- [Error Handling in TypeScript](/how-to/error-handling-patterns-in-typescript) — handling budget errors
- [Google Gemini example (TypeScript)](https://github.com/runcycles/cycles-client-typescript/tree/main/examples/google-gemini) — runnable Google Gemini integration



# Integrating Cycles with Groq

This guide shows how to add budget governance to [Groq](https://groq.com/) API calls. Groq provides an OpenAI-compatible Chat Completions API, so the examples use the OpenAI SDK with a different `base_url`. Confirm Groq support before copying provider-specific OpenAI features beyond that shared surface.

## Prerequisites

```bash
pip install runcycles openai
```

```bash
export CYCLES_BASE_URL="http://localhost:7878"
export CYCLES_API_KEY="your-api-key"   # create via Admin Server — see note below
export CYCLES_TENANT="acme"
export GROQ_API_KEY="gsk_..."
```

> **Need a Cycles API key?** Create one via the Admin Server — see [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack#step-3-create-an-api-key) or [API Key Management](/how-to/api-key-management-in-cycles).

::: tip 60-Second Quick Start
```python
from openai import OpenAI
from runcycles import CyclesClient, CyclesConfig, cycles, set_default_client

set_default_client(CyclesClient(CyclesConfig.from_env()))
groq = OpenAI(base_url="https://api.groq.com/openai/v1", api_key="gsk_...")

@cycles(estimate=100_000, action_kind="llm.completion", action_name="openai/gpt-oss-120b")
def ask(prompt: str) -> str:
    return groq.chat.completions.create(
        model="openai/gpt-oss-120b",
        messages=[{"role": "user", "content": prompt}],
    ).choices[0].message.content

print(ask("What is budget authority?"))
```
Same SDK shape, same `@cycles` decorator, different `base_url`. Size the estimate from the selected Groq model's current price and a conservative token ceiling.
:::

## Basic pattern

```python
import os
from openai import OpenAI
from runcycles import (
    CyclesConfig, CyclesClient, CyclesMetrics,
    cycles, get_cycles_context, set_default_client,
)

set_default_client(CyclesClient(CyclesConfig.from_env()))

groq = OpenAI(
    base_url="https://api.groq.com/openai/v1",
    api_key=os.environ["GROQ_API_KEY"],
)

# GPT-OSS 120B on Groq, checked 2026-07-24:
# $0.15 / 1M input tokens and $0.60 / 1M output tokens.
PRICE_PER_INPUT_TOKEN = 15
PRICE_PER_OUTPUT_TOKEN = 60

@cycles(
    estimate=lambda prompt, **kw: len(prompt.split()) * 2 * PRICE_PER_INPUT_TOKEN
        + kw.get("max_tokens", 1024) * PRICE_PER_OUTPUT_TOKEN,
    actual=lambda result: (
        result["usage"]["prompt_tokens"] * PRICE_PER_INPUT_TOKEN
        + result["usage"]["completion_tokens"] * PRICE_PER_OUTPUT_TOKEN
    ),
    action_kind="llm.completion",
    action_name="openai/gpt-oss-120b",
    unit="USD_MICROCENTS",
)
def chat(prompt: str, max_tokens: int = 1024) -> dict:
    ctx = get_cycles_context()
    if ctx and ctx.has_caps() and ctx.caps.max_tokens:
        max_tokens = min(max_tokens, ctx.caps.max_tokens)

    response = groq.chat.completions.create(
        model="openai/gpt-oss-120b",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
    )

    if ctx:
        ctx.metrics = CyclesMetrics(
            tokens_input=response.usage.prompt_tokens,
            tokens_output=response.usage.completion_tokens,
            model_version=response.model,
        )

    return {
        "content": response.choices[0].message.content,
        "usage": {
            "prompt_tokens": response.usage.prompt_tokens,
            "completion_tokens": response.usage.completion_tokens,
        },
    }
```

## TypeScript

```typescript
import OpenAI from "openai";
import { CyclesClient, CyclesConfig, withCycles, getCyclesContext } from "runcycles";

const cycles = new CyclesClient(CyclesConfig.fromEnv());
const groq = new OpenAI({
  baseURL: "https://api.groq.com/openai/v1",
  apiKey: process.env.GROQ_API_KEY,
});

const INPUT_PRICE = 15;
const OUTPUT_PRICE = 60;

const chat = withCycles(
  {
    client: cycles,
    actionKind: "llm.completion",
    actionName: "openai/gpt-oss-120b",
    estimate: (prompt: string) => {
      const inputTokens = Math.ceil(prompt.length / 4);
      return inputTokens * INPUT_PRICE + 1024 * OUTPUT_PRICE;
    },
    actual: (r: OpenAI.ChatCompletion) =>
      (r.usage?.prompt_tokens ?? 0) * INPUT_PRICE +
      (r.usage?.completion_tokens ?? 0) * OUTPUT_PRICE,
  },
  async (prompt: string) => {
    const ctx = getCyclesContext();
    let maxTokens = 1024;
    if (ctx?.caps?.maxTokens) {
      maxTokens = Math.min(maxTokens, ctx.caps.maxTokens);
    }

    return groq.chat.completions.create({
      model: "openai/gpt-oss-120b",
      max_tokens: maxTokens,
      messages: [{ role: "user", content: prompt }],
    });
  },
);
```

## Groq pricing reference

Groq's on-demand list prices were rechecked on July 24, 2026:

| Model | Input (per 1M tokens) | Output (per 1M tokens) | Input (microcents/1K tokens) | Output (microcents/1K tokens) |
|---|---|---|---|---|
| `openai/gpt-oss-20b` | $0.075 | $0.30 | 7,500 | 30,000 |
| `openai/gpt-oss-120b` | $0.15 | $0.60 | 15,000 | 60,000 |
| `qwen/qwen3.6-27b` | $0.60 | $3.00 | 60,000 | 300,000 |

::: info Note
Groq pricing and model availability change. Check [Groq pricing](https://groq.com/pricing) and [model deprecations](https://console.groq.com/docs/deprecations) before deploying. Llama 4 Scout shut down for free and developer tiers on July 17, 2026; Llama 3.1 8B and Llama 3.3 70B are scheduled to shut down for those tiers on August 16, 2026.
:::

## Model-downgrade degradation pattern

An application can try a lower-estimate Groq route after the primary route's reservation is rejected. Cycles does not choose the fallback or detect a “low budget” threshold automatically; the application owns that policy.

```python
from runcycles import BudgetExceededError

# Primary provider route
primary_client = OpenAI()
PRIMARY_MODEL = os.environ["PRIMARY_MODEL"]

# Lower-estimate Groq route
fallback_client = OpenAI(
    base_url="https://api.groq.com/openai/v1",
    api_key=os.environ["GROQ_API_KEY"],
)

@cycles(
    estimate=1_500_000,
    action_kind="llm.completion",
    action_name="primary-model-route",
)
def primary_chat(prompt: str) -> dict:
    response = primary_client.chat.completions.create(
        model=PRIMARY_MODEL,
        messages=[{"role": "user", "content": prompt}],
    )
    return {"content": response.choices[0].message.content, "model": PRIMARY_MODEL}

@cycles(
    estimate=100_000,
    action_kind="llm.completion",
    action_name="openai/gpt-oss-120b",
)
def fallback_chat(prompt: str) -> dict:
    response = fallback_client.chat.completions.create(
        model="openai/gpt-oss-120b",
        messages=[{"role": "user", "content": prompt}],
    )
    return {"content": response.choices[0].message.content, "model": "openai/gpt-oss-120b"}

def chat_with_downgrade(prompt: str) -> dict:
    """Try the primary route, then a lower-estimate Groq route."""
    try:
        return primary_chat(prompt)
    except BudgetExceededError:
        return fallback_chat(prompt)
```

This pattern gives you:
- **An application-owned fallback** after the primary reservation is rejected
- **A separately estimated Groq route** that may still fit the remaining ledger
- **Per-model attribution** in reservation records through distinct `action_name` values

See [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) for more strategies.

## Key points

- **Same SDK, different `base_url`.** Groq uses the OpenAI-compatible API — no new SDK to learn.
- **Model-specific estimates.** Calculate from current Groq list or contracted rates; do not copy an estimate from a different model.
- **Fallback is application policy.** A rejected primary reservation can trigger a separately budgeted Groq route.
- **Compatibility has limits.** The OpenAI-compatible Chat Completions shape enables shared client code, but verify streaming, tool, and response-field behavior for the selected Groq model.

## Next steps

- [Integrating with OpenAI](/how-to/integrating-cycles-with-openai) — related OpenAI SDK lifecycle patterns
- [Integrating with OpenAI (TypeScript)](/how-to/integrating-cycles-with-openai-typescript) — TypeScript streaming patterns
- [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) — model downgrade and other strategies
- [Cost Estimation Cheat Sheet](/how-to/cost-estimation-cheat-sheet) — pricing reference for all providers
- [Integrating with Ollama](/how-to/integrating-cycles-with-ollama) — self-hosted open-source models



# Integrating Cycles with LangChain.js

Use the TypeScript SDK's lifecycle helpers instead of a raw LangChain callback:

| Workload | Helper |
|---|---|
| One chain/model invocation with a final result | `withCycles` |
| Streaming response or multi-step agent run | `reserveForStream` |

Both paths reserve before execution. Lifecycle-managed commits use the SDK's
durable journal; `reserveForStream` also heartbeats the lease while the run is
active. Version 0.4.3 also prevents broad error cleanup from releasing known
spend after a terminal commit rejection.

## Install

```bash
npm install runcycles@^0.4.3 @langchain/openai @langchain/core
```

## Guard a chain with `withCycles`

```typescript
import { ChatOpenAI } from "@langchain/openai";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import type { AIMessage } from "@langchain/core/messages";
import { BudgetExceededError, CyclesClient, CyclesConfig, withCycles } from "runcycles";

const client = new CyclesClient(CyclesConfig.fromEnv());
const model = new ChatOpenAI({ model: "gpt-4o" });
const prompt = ChatPromptTemplate.fromMessages([["user", "{question}"]]);
const chain = prompt.pipe(model);

function calculatedCost(message: AIMessage): number {
  const usage = message.usage_metadata;
  if (!usage) throw new Error("LangChain returned no normalized usage_metadata");
  const cached = usage.input_token_details?.cache_read ?? 0;
  const ordinaryInput = Math.max(0, usage.input_tokens - cached);
  return ordinaryInput * 250 + cached * 125 + usage.output_tokens * 1_000;
}

const ask = withCycles(
  {
    client,
    actionKind: "llm.completion",
    actionName: "gpt-4o",
    estimate: 2_000_000,
    actual: calculatedCost,
  },
  async (question: string) => chain.invoke({ question }),
);

try {
  const answer = await ask("What is runtime authority?");
  console.log(answer.content);
} catch (error) {
  if (error instanceof BudgetExceededError) {
    console.error("Denied before the model call");
  } else {
    throw error;
  }
}
```

The rates above are caller-supplied examples, not live pricing. Verify the exact
model and cache rate you deploy.

## Streaming and multi-step agents

```typescript
import { HumanMessage } from "@langchain/core/messages";
import { reserveForStream } from "runcycles";

const streamEstimate = 4_000_000;
const handle = await reserveForStream({
  client,
  estimate: streamEstimate,
  unit: "USD_MICROCENTS",
  actionKind: "agent.run",
  actionName: "support-agent",
});

let dispatchAttempted = false;
try {
  // From this point onward the provider may incur partial usage even if the
  // stream throws before returning a final normalized usage object.
  dispatchAttempted = true;
  const stream = await model.stream([new HumanMessage("Draft a reply.")]);
  let finalUsage;
  for await (const chunk of stream) {
    process.stdout.write(typeof chunk.content === "string" ? chunk.content : "");
    if (chunk.usage_metadata) finalUsage = chunk.usage_metadata;
  }
  if (!finalUsage) throw new Error("No finalized normalized usage_metadata");

  const cached = finalUsage.input_token_details?.cache_read ?? 0;
  const actual = Math.max(0, finalUsage.input_tokens - cached) * 250
    + cached * 125
    + finalUsage.output_tokens * 1_000;
  await handle.commit(actual, {
    tokensInput: finalUsage.input_tokens,
    tokensOutput: finalUsage.output_tokens,
  });
} catch (error) {
  if (!handle.finalized) {
    if (dispatchAttempted) {
      // No final usage may exist for an interrupted stream. Conservatively
      // settle the estimate and mark it rather than returning budget for
      // possible partial provider spend.
      try {
        await handle.commit(streamEstimate, undefined, { actual_source: "estimate" });
      } catch (settlementError) {
        console.error("Cycles settlement failed after stream error", settlementError);
      }
    } else {
      await handle.release("stream_startup_failed");
    }
  }
  throw error;
}
```

For a multi-step agent, accumulate normalized usage across its finalized model
messages, apply `handle.caps` before execution, then commit the aggregate once.
See the runnable
[`examples/langchain-js`](https://github.com/runcycles/cycles-client-typescript/tree/main/examples/langchain-js)
project.

## Denial and failure semantics

- Lifecycle helpers throw `BudgetExceededError` before LangChain runs.
- At the raw reserve endpoint, budget denial is HTTP 409
  `BUDGET_EXCEEDED`—not a 2xx `decision: "DENY"` response.
- A pre-dispatch handler error releases. After provider dispatch, interrupted
  streams conservatively commit the estimate when final usage is unavailable.
- Once actual spend is known, commit recovery is journaled and uses the same
  idempotency key. A recognized terminal commit rejection is surfaced with the
  handle finalized; never release known spend from a broad catch.
- A Cycles key deduplicates accounting only. Keep tool side effects separately
  idempotent.

## Why not the old callback recipe?

A simple `handleLLMStart`/`handleLLMEnd` map does not automatically heartbeat a
long call or durably persist a pending commit before process exit. It can also
read provider-specific `llmOutput.tokenUsage` while claiming provider-neutral
behavior. Use the SDK helpers above unless you implement equivalent lease and
recovery choreography yourself.

## Next steps

- [TypeScript error handling](/how-to/error-handling-patterns-in-typescript)
- [Handling streaming responses](/how-to/handling-streaming-responses-with-cycles)
- [Cost estimation cheat sheet](/how-to/cost-estimation-cheat-sheet)



# LangChain Budget Control with Cycles

Use [`langchain-runcycles`](https://pypi.org/project/langchain-runcycles/) for
LangChain 1.x agents built with `langchain.agents.create_agent`. It provides
three `AgentMiddleware` controls:

| Middleware | Boundary |
|---|---|
| `CyclesFanOutGate` | Stops another model turn when a local cap or remote policy says stop |
| `CyclesModelGate` | Authorizes and optionally reserves before each model call |
| `CyclesToolGate` | Authorizes and optionally reserves before each tool side effect |

For bare models, chains, and RAG runnables that do not use `create_agent`, use
the managed callback example in the
[`runcycles` Python SDK](https://github.com/runcycles/cycles-client-python/blob/main/examples/langchain_integration.py).

## Install

```bash
pip install "langchain-runcycles>=0.4.0" langchain-anthropic
```

Version 0.4.0 requires `runcycles >=0.5.3` and uses the SDK's managed
reservation lifecycle. Reserve-mode calls are heartbeated while the handler
runs, and known spend is journaled before the first commit request.

## Compose the three gates

```python
from langchain.agents import create_agent
from langchain.tools import tool
from langchain_runcycles import CyclesFanOutGate, CyclesModelGate, CyclesToolGate
from langchain_runcycles.extractors import anthropic_cost
from runcycles import Action, Amount, CyclesClient, CyclesConfig, Subject, Unit

client = CyclesClient(CyclesConfig.from_env())
subject = Subject(tenant="acme", workflow="support", agent="researcher")

@tool
def send_email(to: str, body: str) -> str:
    """Send an email after all middleware checks pass."""
    return f"Sent to {to}"

agent = create_agent(
    model="claude-sonnet-4-6",
    tools=[send_email],
    middleware=[
        CyclesFanOutGate(
            max_turns=20,
            client=client,
            subject=subject,
            action=Action(kind="model.turn", name="support"),
        ),
        CyclesModelGate(
            client,
            subject=subject,
            action=Action(kind="llm.completion", name="claude-sonnet-4-6"),
            mode="decide+reserve",
            estimate=Amount(unit=Unit.USD_MICROCENTS, amount=2_500_000),
            cost_fn=anthropic_cost(
                input_per_million_usd=3.00,
                output_per_million_usd=15.00,
                cache_read_per_million_usd=0.30,
                cache_creation_5m_per_million_usd=3.75,
                cache_creation_1h_per_million_usd=6.00,
            ),
        ),
        CyclesToolGate(
            client,
            subject=subject,
            action={"send_email": Action(kind="tool.call", name="send_email")},
            mode="decide+reserve",
            idempotency_namespace=lambda request: request.state.get("run_id"),
        ),
    ],
)

agent.invoke({
    "messages": [{"role": "user", "content": "Email the customer."}],
    "run_id": "support-run-123",
})
```

The order is intentional: stop runaway fan-out, authorize model spend, then
authorize tool side effects.

## Choose a gate mode

`CyclesModelGate` and `CyclesToolGate` share three modes:

| Mode | Behavior |
|---|---|
| `"decide"` | Policy/budget preflight only; no hold or settlement |
| `"reserve"` | Reserve before the handler, commit after success, release on handler failure |
| `"decide+reserve"` | Run both checks; strongest separation of policy and accounting |

Both `ALLOW` and `ALLOW_WITH_CAPS` are allowed decisions. A tool denial is
returned as a correlated `ToolMessage`; a model denial becomes a
`ModelResponse` that ends the loop. At the raw protocol level, an exhausted
reservation request is HTTP 409 `BUDGET_EXCEEDED`, not a successful 2xx body.

## Settlement after the action ran

Reserve mode uses the same recovery choreography as the Python SDK:

1. heartbeat the reservation while the model/tool handler runs;
2. persist the exact known-spend commit before sending it;
3. replay transient failures with the same key, including after restart;
4. if the reservation expires, recover through `POST /v1/events`.

`settlement_error_policy` controls what the current LangChain call observes,
not whether recovery exists:

| Policy | Result |
|---|---|
| `"raise"` (default) | Queue recovery, then raise `CyclesProtocolError` |
| `"log"` | Queue recovery, log, and return the handler result |

Use `"log"` for non-idempotent side effects when an automatic agent retry could
repeat an email, payment, or write. Known spend remains recoverable either way.

## Actual-cost extraction

The model extractors read LangChain's provider-neutral
`AIMessage.usage_metadata`. Their rates are caller supplied and are not a live
pricing service:

```python
from langchain_runcycles.extractors import openai_cost

cost_fn = openai_cost(
    prompt_per_million_usd=2.50,
    cached_prompt_per_million_usd=1.25,
    completion_per_million_usd=10.00,
)
```

Cache reads/writes come from normalized `input_token_details`. Verify the exact
model and cache tier on the provider pricing page before deployment. A malformed
usage object, invalid rate, or mismatched unit falls back to the configured
reservation estimate.

Tool providers do not expose one normalized billing shape. Supply
`CyclesToolGate.cost_fn(request, result)` when a tool should commit something
other than its configured estimate.

## Idempotency scope

Tool reservation keys are stable when LangChain provides `tool_call_id`.
Configure `idempotency_namespace` when short call IDs might repeat across runs:

```python
tool_gate = CyclesToolGate(
    client,
    subject=subject,
    action=Action(kind="tool.call", name="send_email"),
    mode="reserve",
    idempotency_namespace=lambda request: request.state["run_id"],
)
```

If a tool ID is missing, the middleware generates a logged random fallback.
Model and fan-out hooks have no equivalent stable upstream call ID and use a
fresh UUID within the optional namespace; do not assume those keys survive a
framework redispatch.

## Async and streaming

Pass `AsyncCyclesClient` and call `.ainvoke()` for async hooks. Completed
`agent.astream(...)` and `agent.astream_events(...)` runs are heartbeated and
settled once from LangChain's final aggregated `usage_metadata`.

If a stream is cancelled before LangChain produces a final response, no
finalized normalized usage exists. The middleware releases the reservation and
re-raises; reconcile any provider charge for the partial stream from provider
billing telemetry.

## Non-agent callbacks

The SDK's
[`CyclesBudgetHandler`](https://github.com/runcycles/cycles-client-python/blob/main/examples/langchain_integration.py)
recipe keeps one managed reservation per LangChain `run_id`, uses a lock for
concurrent callbacks, reads normalized usage (including cache reads), and
settles through the SDK journal. Avoid low-level callback examples that simply
pop an in-memory reservation and ignore the commit response.

## Next steps

- [Add budget control to a LangChain agent](/how-to/how-to-add-budget-control-to-a-langchain-agent)
- [Integrate Cycles with LangGraph](/how-to/integrating-cycles-with-langgraph)
- [`langchain-runcycles` source and audit](https://github.com/runcycles/langchain-runcycles)
- [SDK settlement recovery and durability](/protocol/sdk-settlement-recovery-and-durability)



# Integrating Cycles with LangGraph

Choose the integration surface that owns the execution boundary:

| Graph style | Recommended Cycles surface |
|---|---|
| Agent built with `langchain.agents.create_agent` | `CyclesFanOutGate` + `CyclesModelGate` + `CyclesToolGate` |
| Raw `StateGraph` node | Python `@cycles` or `stream_reservation()` around the node's paid work |
| Conditional policy edge | `client.decide()` and accept both `ALLOW` and `ALLOW_WITH_CAPS` |

Do not start new code with the deprecated `langgraph.prebuilt.create_react_agent`.
LangChain's current agent entry point is `langchain.agents.create_agent`.

## Agent graphs

```python
from langchain.agents import create_agent
from langchain_runcycles import CyclesFanOutGate, CyclesModelGate, CyclesToolGate
from runcycles import Action, Amount, CyclesClient, CyclesConfig, Subject, Unit

client = CyclesClient(CyclesConfig.from_env())
subject = Subject(tenant="acme", workflow="research")

agent = create_agent(
    model="claude-sonnet-4-6",
    tools=[search, publish],
    middleware=[
        CyclesFanOutGate(
            max_turns=20,
            client=client,
            subject=subject,
            action=Action(kind="model.turn", name="research"),
        ),
        CyclesModelGate(
            client,
            subject=subject,
            action=Action(kind="llm.completion", name="claude-sonnet-4-6"),
            mode="reserve",
            estimate=Amount(unit=Unit.USD_MICROCENTS, amount=2_500_000),
        ),
        CyclesToolGate(
            client,
            subject=subject,
            action={
                "search": Action(kind="tool.call", name="search"),
                "publish": Action(kind="tool.call", name="publish"),
            },
            mode="decide+reserve",
            idempotency_namespace=lambda request: request.state["run_id"],
        ),
    ],
)
```

Reserve modes heartbeat long agent work and durably settle known spend. Tool
keys are stable when LangChain supplies `tool_call_id`; model/fan-out calls do
not have that upstream stable identity and use a fresh UUID within the optional
namespace.

## Raw StateGraph nodes

Use the SDK lifecycle instead of hand-writing reserve/commit/release inside a
`try` block. The decorator provides heartbeat and durable recovery:

```python
from langgraph.graph import StateGraph
from runcycles import cycles

def actual_cost(result: dict) -> int:
    message = result["messages"][-1]
    usage = message.usage_metadata or {}
    return usage.get("input_tokens", 0) * 250 + usage.get("output_tokens", 0) * 1_000

@cycles(
    client=client,
    tenant="acme",
    workflow="research",
    action_kind="llm.completion",
    action_name="gpt-4o",
    estimate=2_000_000,
    actual=actual_cost,
)
def call_model(state: dict) -> dict:
    message = model.invoke(state["messages"])
    return {"messages": [message]}

graph = StateGraph(dict)
graph.add_node("model", call_model)
```

If you need per-run values, make the decorator's estimate/subject fields
callables or build the managed reservation inside the node. Do not release in a
catch block around commit: once the model ran, that can return budget for spend
that already happened.

## Conditional policy edges

```python
from runcycles import Action, Amount, DecisionRequest, DecisionResponse, Unit

def route(state: dict) -> str:
    response = client.decide(DecisionRequest(
        idempotency_key=f"graph-route-{state['run_id']}-{state['step']}",
        subject=subject,
        action=Action(kind="graph.step", name="continue"),
        estimate=Amount(unit=Unit.RISK_POINTS, amount=1),
    ))
    if not response.is_success or response.body is None:
        return "end"
    decision = DecisionResponse.model_validate(response.body)
    return "continue" if decision.is_allowed() else "end"
```

`DecisionResponse.is_allowed()` treats both `ALLOW` and `ALLOW_WITH_CAPS` as
permitted. Apply any returned cap fields in the node that owns the constrained
operation.

## Durable graph retries versus Cycles idempotency

LangGraph checkpoint replay and Cycles accounting idempotency solve different
problems. A stable Cycles key prevents duplicate accounting; it does not make an
email, payment, or database write consume-once. Store run/step IDs in graph
state, scope tool keys with them, and keep external side effects independently
idempotent.

## Next steps

- [LangChain agent middleware guide](/how-to/integrating-cycles-with-langchain)
- [LangGraph durable execution, retries, and fan-out](/blog/langgraph-budget-control-durable-execution-retries-fan-out)
- [SDK settlement recovery and durability](/protocol/sdk-settlement-recovery-and-durability)



# Integrating Cycles with LlamaIndex

This guide shows how to guard LlamaIndex RAG queries with Cycles budget reservations so that every retrieval and generation call is cost-controlled and observable.

## Prerequisites

```bash
pip install runcycles llama-index
```

```bash
export CYCLES_BASE_URL="http://localhost:7878"
export CYCLES_API_KEY="your-api-key"   # create via Admin Server — see note below
export CYCLES_TENANT="acme"
export OPENAI_API_KEY="sk-..."
```

> **Need an API key?** Create one via the Admin Server — see [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack#step-3-create-an-api-key) or [API Key Management](/how-to/api-key-management-in-cycles).

::: tip 60-Second Quick Start
```python
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from runcycles import CyclesClient, CyclesConfig, cycles, set_default_client

set_default_client(CyclesClient(CyclesConfig.from_env()))

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()

@cycles(estimate=2_000_000, action_kind="rag.query", action_name="llamaindex-query")
def ask(question: str) -> str:
    response = query_engine.query(question)
    return str(response)

print(ask("What are the key findings?"))
```
Every query is now budget-guarded. If the budget is exhausted, `BudgetExceededError` is raised _before_ the query executes. Read on for production patterns.
:::

## Guarding index queries

Use the `@cycles` decorator to wrap a query engine call with automatic reserve, execute, and commit:

```python
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from runcycles import (
    CyclesClient, CyclesConfig, CyclesMetrics,
    cycles, get_cycles_context, set_default_client, BudgetExceededError,
)

config = CyclesConfig.from_env()
set_default_client(CyclesClient(config))

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()

PRICE_PER_INPUT_TOKEN = 250       # $2.50 / 1M tokens
PRICE_PER_OUTPUT_TOKEN = 1_000    # $10.00 / 1M tokens

@cycles(
    estimate=lambda question, **kw: len(question.split()) * 4 * PRICE_PER_INPUT_TOKEN
        + 1024 * PRICE_PER_OUTPUT_TOKEN,
    action_kind="rag.query",
    action_name="llamaindex-query",
    unit="USD_MICROCENTS",
    ttl_ms=120_000,
)
def ask(question: str) -> str:
    response = query_engine.query(question)

    ctx = get_cycles_context()
    if ctx:
        ctx.metrics = CyclesMetrics(model_version="gpt-4o")

    return str(response)
```

## Guarding retrieval and generation separately

For fine-grained cost tracking, decorate the retrieval and generation steps independently:

```python
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.llms import ChatMessage
from llama_index.llms.openai import OpenAI
from runcycles import cycles, get_cycles_context, CyclesMetrics

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
retriever = index.as_retriever(similarity_top_k=5)
llm = OpenAI(model="gpt-4o")

@cycles(estimate=100_000, action_kind="tool.search", action_name="vector-retrieval")
def retrieve(question: str) -> list:
    return retriever.retrieve(question)

@cycles(
    estimate=2_000_000,
    action_kind="llm.completion",
    action_name="gpt-4o",
    unit="USD_MICROCENTS",
)
def generate(question: str, context_nodes: list) -> str:
    context_text = "\n".join(node.get_content() for node in context_nodes)
    prompt = f"Context:\n{context_text}\n\nQuestion: {question}"
    response = llm.chat([ChatMessage(role="user", content=prompt)])

    ctx = get_cycles_context()
    if ctx:
        ctx.metrics = CyclesMetrics(
            tokens_input=response.raw.usage.prompt_tokens,
            tokens_output=response.raw.usage.completion_tokens,
            model_version="gpt-4o",
        )

    return str(response)

# Pipeline: retrieve then generate, each independently budget-guarded
nodes = retrieve("What are the key findings?")
answer = generate("What are the key findings?", nodes)
```

## Cost estimation for RAG pipelines

RAG pipelines involve both retrieval (embedding lookups) and generation (LLM calls). Estimate each stage separately for accuracy:

| Stage | action_kind | Estimation strategy |
|-------|-------------|---------------------|
| Embedding / retrieval | `tool.search` | Flat cost per query (embedding calls are cheap) |
| Generation | `llm.completion` | Input tokens (context + question) + max output tokens |

For production, estimate generation cost based on the retrieved context size:

```python
@cycles(
    estimate=lambda question, context_nodes, **kw: (
        sum(len(n.get_content().split()) for n in context_nodes) * 2 * PRICE_PER_INPUT_TOKEN
        + 1024 * PRICE_PER_OUTPUT_TOKEN
    ),
    action_kind="llm.completion",
    action_name="gpt-4o",
)
def generate_with_context(question: str, context_nodes: list) -> str:
    context_text = "\n".join(node.get_content() for node in context_nodes)
    prompt = f"Context:\n{context_text}\n\nQuestion: {question}"
    return str(llm.chat([ChatMessage(role="user", content=prompt)]))
```

## Error handling

When the budget is insufficient, `BudgetExceededError` is raised **before** the query executes:

```python
from runcycles import BudgetExceededError

try:
    answer = ask("Summarize the entire dataset...")
except BudgetExceededError:
    answer = "Budget limit reached. Please try a shorter query or contact your administrator."
```

For retrieval-then-generation pipelines, handle each step:

```python
try:
    nodes = retrieve(question)
    answer = generate(question, nodes)
except BudgetExceededError:
    answer = "Service temporarily unavailable due to budget limits."
```

See [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) for patterns like caching, model downgrade, and queueing.

## Key points

- **Wrap any function.** The `@cycles` decorator works on any callable, so LlamaIndex query engines, retrievers, and LLM calls all work out of the box.
- **Split retrieval and generation.** Separate decorators give per-stage cost visibility and independent budget control.
- **Estimate before, commit after.** The `estimate` function determines the reservation; actual cost is committed after execution.
- **The function never executes on DENY.** Neither the retrieval nor the LLM call runs if the budget is exhausted.

## Next steps

- [Error Handling Patterns in Python](/how-to/error-handling-patterns-in-python) — handling budget errors in Python
- [Testing with Cycles](/how-to/testing-with-cycles) — testing budget-guarded code
- [Integrating with LangChain](/how-to/integrating-cycles-with-langchain) — budget governance for LangChain apps
- [Production Operations Guide](/how-to/production-operations-guide) — running Cycles in production



# Integrating Cycles with MCP

::: tip Put budget checks in the execution path
For MCP tool handlers, read [Add Hard Budgets to MCP Tools Before They Execute](/blog/mcp-tool-budgets-before-execution) for a TypeScript reserve-commit wrapper. On Claude Code, [Cycles Budget Guard for Claude Code](/how-to/enforcing-budgets-in-claude-code-with-budget-guard) gates non-exempt tools at the harness layer instead of relying on the model to cooperate.
:::

The [Model Context Protocol](https://modelcontextprotocol.io) (MCP) is the standard way AI hosts discover and call tools. The Cycles MCP Server exposes Cycles runtime authority as MCP tools, so MCP-compatible agents (Claude Desktop, Claude Code, Cursor, Windsurf, custom agents) can call `decide`, `reserve`, `commit`, `release`, and balance tools without an SDK integration.

This gives agents a standard way to participate in Cycles workflows. **For hard budget enforcement, make the Cycles check part of the actual execution path: the tool call, model call, gateway, or harness must require a successful live `reserve` before the costly action fires.** `decide` is a non-locking preflight whose result the application must apply. The MCP server alone exposes tools; it does not automatically gate every other action the agent might take.

This guide covers the integration patterns, resources, prompts, and transport options available through the MCP server.

## Choose your path

- **Cap MCP tool calls in JS/TS** — install [`@runcycles/mcp-server`](https://www.npmjs.com/package/@runcycles/mcp-server) and follow this guide.
- **Run the full Cycles stack locally first** — bring up [server + admin + dashboard](/quickstart/deploying-the-full-cycles-stack) before wiring up MCP, so you can see denials in the dashboard while testing.
- **Evaluate Cycles for a multi-tenant agent SaaS** — start with the [evaluation guide](/how-to/evaluate-cycles-for-agent-saas) before reading this implementation page.
- **Not sure where Cycles fits?** [Send us your tool-call flow](/contact) and we'll map where `reserve` / `commit` should sit.

::: tip No SDK changes
MCP integration requires no SDK changes in your agent application. You configure the Cycles MCP Server in your host, and the agent can discover Cycles tools.

For deterministic enforcement, do not rely on the model voluntarily calling these tools. Put the Cycles check in the tool execution path or gateway layer.
:::

## Prerequisites

```bash
npm install @runcycles/mcp-server   # or use npx at runtime
```

```bash
export CYCLES_API_KEY="cyc_live_..."    # from Admin Server
export CYCLES_BASE_URL="http://localhost:7878"  # required — your Cycles server URL
```

For local development without an API key:

```bash
export CYCLES_MOCK=true
```

> **Need setup help?** See [Getting Started with the MCP Server](/quickstart/getting-started-with-the-mcp-server) for per-host configuration (Claude Desktop, Claude Code, Cursor, Windsurf).

## Pattern 1: Simple reserve-commit

The most common pattern — reserve budget before a costly operation, commit actual usage after:

**Step 1 — Reserve:**

```json
{
  "idempotencyKey": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "subject": { "tenant": "acme", "agent": "researcher" },
  "action": { "kind": "llm.completion", "name": "claude-sonnet" },
  "estimate": { "unit": "USD_MICROCENTS", "amount": 50000 },
  "ttlMs": 60000
}
```

An accepted response includes `decision: "ALLOW"` or `decision: "ALLOW_WITH_CAPS"` and a `reservationId`. Apply any returned caps before executing. A live denial is returned as an MCP tool error, such as `BUDGET_EXCEEDED`, rather than as `decision: "DENY"`.

**Step 2 — Execute** the LLM call or tool invocation.

**Step 3 — Commit:**

```json
{
  "reservationId": "rsv_...",
  "idempotencyKey": "commit-a1b2c3d4",
  "actual": { "unit": "USD_MICROCENTS", "amount": 35000 },
  "metrics": {
    "tokensInput": 1200,
    "tokensOutput": 800,
    "latencyMs": 2500,
    "modelVersion": "claude-sonnet-4-6"
  }
}
```

The unused 15,000 microcents are returned to the budget pool.

**If the operation never starts and incurs no usage**, call `cycles_release` instead. If execution starts and incurs partial usage before failing, call `cycles_commit` with that actual usage so the cost is not lost:

```json
{
  "reservationId": "rsv_...",
  "idempotencyKey": "release-a1b2c3d4",
  "reason": "Operation cancelled before dispatch"
}
```

## Pattern 2: Preflight + reserve

Use `cycles_decide` for a lightweight check before committing to a reservation. Useful at the start of a workflow to decide strategy:

```json
{
  "idempotencyKey": "decide-uuid",
  "subject": { "tenant": "acme", "workflow": "summarize" },
  "action": { "kind": "llm.completion", "name": "claude-opus" },
  "estimate": { "unit": "USD_MICROCENTS", "amount": 200000 }
}
```

If the decision is `ALLOW`, proceed with a full `cycles_reserve`. If it is `ALLOW_WITH_CAPS`, apply the returned caps and reserve the constrained estimate. If it is `DENY`, the agent can switch to a cheaper model or skip the operation — without having locked any budget.

## Pattern 3: Graceful degradation

When the deepest matching budget has caps configured, `cycles_reserve` can return `ALLOW_WITH_CAPS` instead of a flat `ALLOW`. This is configuration-driven; it is not an automatic low-balance transition. Caps tell the caller how to constrain the operation:

```json
{
  "decision": "ALLOW_WITH_CAPS",
  "reservationId": "rsv_...",
  "caps": {
    "maxTokens": 2000,
    "toolDenylist": ["web_search", "code_execution"],
    "cooldownMs": 5000
  }
}
```

The `caps` payload may include hints such as max tokens, allowed or denied tools, remaining steps, or cooldown timing. Common fields the agent should respect when present:

- max output tokens on the LLM call
- max remaining agent steps
- allowed / denied tool lists
- cooldown between operations to slow spend rate

See [Caps and the Three-Way Decision Model](/protocol/caps-and-the-three-way-decision-model-in-cycles) for the full schema and current field names.

## Pattern 4: Long-running operations

For operations that may exceed the default 60-second TTL, use `cycles_extend` as a heartbeat:

**Reserve with a TTL:**

```json
{
  "idempotencyKey": "long-op-uuid",
  "subject": { "tenant": "acme", "workflow": "data-pipeline" },
  "action": { "kind": "batch", "name": "process-dataset" },
  "estimate": { "unit": "USD_MICROCENTS", "amount": 500000 },
  "ttlMs": 120000
}
```

**Extend periodically** (e.g., every 60 seconds):

```json
{
  "reservationId": "rsv_...",
  "idempotencyKey": "extend-1-uuid",
  "extendByMs": 120000
}
```

**Commit when done.** If the agent crashes, TTL expiry can recover an abandoned hold, but it is not accurate settlement when execution may have started. Reconcile the outcome and commit the best-known actual usage. If the outcome is ambiguous, do not release the reservation merely to restore budget.

See [TTL, Grace Period, and Extend](/protocol/reservation-ttl-grace-period-and-extend-in-cycles) for the full TTL model.

## Pattern 5: Fire-and-forget events

When you can't pre-estimate cost (e.g., webhook-triggered actions, post-hoc metering), use `cycles_create_event` to record usage directly:

```json
{
  "idempotencyKey": "event-uuid",
  "subject": { "tenant": "acme", "app": "chatbot" },
  "action": { "kind": "llm.completion", "name": "gpt-4o" },
  "actual": { "unit": "USD_MICROCENTS", "amount": 42000 },
  "metrics": {
    "tokensInput": 3000,
    "tokensOutput": 1500,
    "latencyMs": 1800
  }
}
```

No reservation needed — the event is applied atomically to all derived scopes. See [Events and Direct Debit](/protocol/how-events-work-in-cycles-direct-debit-without-reservation).

> **This is post-hoc metering, not pre-execution enforcement.** `cycles_create_event` records that the action happened — it does not stop the action before it happens. For preventative control, use `cycles_decide` (preflight) or `cycles_reserve` (lock budget) before execution.

## Pattern 6: Multi-step workflow

For workflows with multiple costly steps, check the balance first, then reserve per step:

**Check balance:**

```json
{ "tenant": "acme", "workflow": "research-report" }
```

**Step 1:** `cycles_reserve` → execute → `cycles_commit`

**Step 2:** `cycles_reserve` → execute → `cycles_commit`

**Step 3:** `cycles_reserve` → **`BUDGET_EXCEEDED` tool error** (budget exhausted) → degrade or stop

Each step gets its own reservation, so the budget authority can deny mid-workflow if the agent is burning through budget too fast. **Do not reserve once for an entire long workflow unless you are comfortable locking that whole estimate up front** — per-step reservations give the authority layer a chance to stop mid-run, and unused budget returns to the pool sooner. See [Common Budget Patterns](/how-to/common-budget-patterns) for more examples.

## Tool reference

The MCP server exposes 9 tools:

| Tool | Description |
|------|-------------|
| `cycles_reserve` | Create a budget reservation before executing a costly operation |
| `cycles_commit` | Finalize a reservation with actual usage |
| `cycles_release` | Release an unused reservation back to the budget pool |
| `cycles_extend` | Extend the TTL of an active reservation (heartbeat) |
| `cycles_decide` | Lightweight budget check without creating a reservation |
| `cycles_create_event` | Record usage directly without a reservation (post-hoc metering) |
| `cycles_check_balance` | Query current budget balance for a tenant/scope |
| `cycles_list_reservations` | List reservations with optional status and subject filters |
| `cycles_get_reservation` | Get details of a specific reservation by ID |

## Resources

The MCP server exposes resources for inspecting budget state:

| URI | Description |
|-----|-------------|
| `cycles://balances/{tenant}` | Current budget balance for a tenant scope |
| `cycles://reservations/{reservation_id}` | Reservation details by ID |
| `cycles://docs/quickstart` | Getting started guide |
| `cycles://docs/patterns` | Integration patterns reference |

Use resources when you need to inspect state without calling a tool — for example, reading a tenant's balance as context before deciding on a strategy.

## Prompts

The server ships three prompts that help AI assistants work with Cycles:

### `integrate_cycles`

Generates Cycles integration code for a given language and use case.

| Parameter | Required | Description |
|-----------|----------|-------------|
| `language` | No | Programming language (default: typescript) |
| `use_case` | No | Context: `llm-calls`, `api-gateway`, `multi-agent` |

> "Use the integrate_cycles prompt to generate Python code for an LLM-calls use case"

### `diagnose_overrun`

Guides through debugging budget exhaustion or a stopped run.

| Parameter | Required | Description |
|-----------|----------|-------------|
| `reservation_id` | No | Specific reservation to investigate |
| `scope` | No | Tenant or scope identifier to check |

> "Use the diagnose_overrun prompt to figure out why my agent stopped — scope is tenant:acme"

### `design_budget_strategy`

Recommends scope hierarchy, budget limits, units, TTL settings, and degradation strategy.

| Parameter | Required | Description |
|-----------|----------|-------------|
| `description` | Yes | Description of the workflow to budget |
| `tenant_model` | No | e.g., `per-customer`, `per-team`, `single-tenant` |

> "Use the design_budget_strategy prompt for my multi-agent customer support system with per-customer tenants"

## Transport modes

The Cycles MCP server supports two transports:

- **STDIO** *(default)* — the AI client launches a local server subprocess. The recommended [Claude Desktop](/quickstart/mcp-claude-desktop) path installs the bundled `.mcpb` extension; manual Desktop configuration and the [Claude Code](/quickstart/mcp-claude-code), [Cursor](/quickstart/mcp-cursor), and [Windsurf](/quickstart/mcp-windsurf) quickstarts launch the npm package with `npx`.
- **Streamable HTTP / SSE compatibility** — the server runs as a long-lived process and clients connect remotely. Streamable HTTP is the current MCP transport; the older standalone HTTP+SSE transport is not implemented. Use this for shared team gateways, cloud co-deploys with `cycles-server`, CI sidecars, or any case where you want auth and audit in front of MCP.

Quick HTTP start:

```bash
HOST=127.0.0.1 \
MCP_HTTP_AUTH_TOKEN='replace-with-a-long-random-secret' \
npx @runcycles/mcp-server --transport http
```

This keeps the listener on the local machine and requires a bearer token for `/mcp`. The server does not currently validate the HTTP `Origin` header, so place remote deployments behind a trusted reverse proxy that validates origins, terminates TLS, and applies authorization.

The server starts on port 3000 (configurable via `PORT`) with:

- `GET /health` — health check (`{"status": "ok", "version": "..."}`)
- `POST /mcp` — MCP Streamable HTTP endpoint
- `GET /mcp` — Streamable HTTP SSE stream (server-to-client notifications)
- `DELETE /mcp` — requests Streamable HTTP transport termination; authenticate and restrict it like the other `/mcp` methods

For the full decision tree, docker-compose example, and auth/scope behavior, see **[Running the MCP server over Streamable HTTP / SSE](/how-to/running-the-mcp-server-over-http)**.

## Error handling

Errors from live mutating calls are returned as MCP tool errors containing the Cycles error code, message, request ID, and HTTP status. In particular, a denied live reservation normally surfaces as a 409-class error; `decision: "DENY"` is reserved for `cycles_decide` and dry-run reserve responses.

| Error Code | Meaning | Recommended Action |
|---|---|---|
| `BUDGET_EXCEEDED` | Not enough budget | Degrade to cheaper model or stop |
| `RESERVATION_EXPIRED` | TTL elapsed before commit | Inspect the outcome and reconcile any usage already incurred; re-reserve only for new work |
| `RESERVATION_FINALIZED` | Already committed or released | Read the reservation status and verify the expected settlement; record missing usage idempotently rather than assuming it was charged |
| `DEBT_OUTSTANDING` | Scope has unpaid debt (no overdraft limit) | Wait for admin to fund the budget or configure an overdraft limit |
| `OVERDRAFT_LIMIT_EXCEEDED` | Over-limit state | Wait for admin to reconcile |

See [Error Codes and Error Handling](/protocol/error-codes-and-error-handling-in-cycles) for the full reference.

## Key points

- **No SDK changes for tool exposure.** Add the MCP server to your agent's config and it discovers Cycles tools automatically. Hard enforcement still requires those tools to sit in the execution path.
- **Always finalize reservations.** Commit actual usage when execution incurred cost, even if it later failed. Release only an unused reservation when execution was cancelled, skipped, or failed before starting. Never leave reservations dangling.
- **Use stable idempotency keys.** Use a unique, stable `idempotencyKey` per logical Cycles operation so retries replay safely and do not double-settle reservations. The same retry of the same logical call must use the **same** key, not a new UUID per attempt.
- **Respect caps.** When the decision is `ALLOW_WITH_CAPS`, constrain the operation accordingly.
- **Heartbeat long operations.** Use `cycles_extend` for operations that may exceed the reservation TTL.
- **Tag for observability.** Use `action.tags` and `metrics.custom` to add context for debugging and auditing.

## Next steps

- [Getting Started with the MCP Server](/quickstart/getting-started-with-the-mcp-server) — setup guide for each AI host
- [Architecture Overview](/quickstart/architecture-overview-how-cycles-fits-together) — how the MCP server fits into the Cycles stack
- [Cost Estimation Cheat Sheet](/how-to/cost-estimation-cheat-sheet) — pricing reference for estimates
- [Troubleshooting and FAQ](/how-to/troubleshooting-and-faq) — common issues and solutions

## Related concepts

- [What is runtime authority?](/blog/what-is-runtime-authority-for-ai-agents)
- [Graceful degradation patterns](/blog/when-budget-runs-out-graceful-degradation-patterns-for-ai-agents)
- [Multi-agent coordination failure: structural prevention](/blog/multi-agent-coordination-failure-structural-prevention)



# Integrating Cycles with Next.js

This guide shows how to add budget governance to a Next.js application using API routes, server actions, and client-side error handling.

For streaming patterns with the Vercel AI SDK, see [Integrating with Vercel AI SDK](/how-to/integrating-cycles-with-vercel-ai-sdk). This guide covers the broader Next.js integration: route-level guards, server actions, per-tenant isolation, and shared client setup.

## Prerequisites

- A running Cycles stack with a tenant, API key, and budget ([Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack))
- A Next.js 15+ project (App Router)
- Node.js 20+

## Installation

```bash
npm install runcycles
```

## Environment variables

Add to `.env.local`:

```bash
CYCLES_BASE_URL=http://localhost:7878
CYCLES_API_KEY=cyc_live_...
CYCLES_TENANT=acme
OPENAI_API_KEY=sk-...
```

## Shared Cycles client

Create a singleton client for use across API routes and server actions:

```typescript
// lib/cycles.ts
import { CyclesClient, CyclesConfig } from "runcycles";

export const cyclesClient = new CyclesClient(CyclesConfig.fromEnv());
```

## Route-level budget guard

Wrap the expensive work in `withCycles` so each request gets an automatic reserve → execute → commit lifecycle (for a lighter `client.decide()` preflight check, see the next section):

```typescript
// app/api/chat/route.ts
import { NextResponse } from "next/server";
import { cyclesClient } from "@/lib/cycles";
import {
  withCycles, getCyclesContext, BudgetExceededError,
} from "runcycles";

export const runtime = "nodejs";

const INPUT_PRICE = 250;    // GPT-4o: $2.50/1M tokens
const OUTPUT_PRICE = 1_000; // GPT-4o: $10/1M tokens

export async function POST(req: Request) {
  const { prompt } = await req.json();

  const generate = withCycles(
    {
      client: cyclesClient,
      actionKind: "llm.completion",
      actionName: "gpt-4o",
      estimate: () => {
        const inputTokens = Math.ceil(prompt.length / 4);
        return inputTokens * INPUT_PRICE + 1024 * OUTPUT_PRICE;
      },
      actual: (result: { usage: { prompt_tokens: number; completion_tokens: number } }) =>
        result.usage.prompt_tokens * INPUT_PRICE +
        result.usage.completion_tokens * OUTPUT_PRICE,
    },
    async () => {
      const ctx = getCyclesContext();
      let maxTokens = 1024;
      if (ctx?.caps?.maxTokens) {
        maxTokens = Math.min(maxTokens, ctx.caps.maxTokens);
      }

      // Your LLM call here
      const response = await callLLM(prompt, maxTokens);

      if (ctx) {
        ctx.metrics = {
          tokensInput: response.usage.prompt_tokens,
          tokensOutput: response.usage.completion_tokens,
        };
      }

      return response;
    },
  );

  try {
    const result = await generate();
    return NextResponse.json({ content: result.content });
  } catch (err) {
    if (err instanceof BudgetExceededError) {
      return NextResponse.json(
        { error: "budget_exceeded", message: "Budget exhausted." },
        { status: 402 },
      );
    }
    throw err;
  }
}
```

::: info 402 is an application-level choice
Returning 402 to the browser is this route's UX decision. At the Cycles protocol layer, budget exhaustion surfaces as a `DENY` decision from `decide()` or a `BudgetExceededError` thrown by `withCycles` — translating either signal to an HTTP status is up to your application.
:::

## Budget preflight in API routes

Next.js middleware runs in the Edge Runtime, which does not support Node.js APIs required by the `runcycles` client. Instead, add a preflight budget check at the start of your API route handler:

```typescript
// app/api/chat/route.ts
import { NextResponse } from "next/server";
import { cyclesClient } from "@/lib/cycles";

export async function POST(req: Request) {
  const tenant = req.headers.get("x-tenant-id") ?? "acme";

  // Preflight: check budget before doing expensive work
  const preflight = await cyclesClient.decide({
    idempotency_key: crypto.randomUUID(),
    subject: { tenant, app: "my-nextjs-app" },
    action: { kind: "api.request", name: "/api/chat" },
    estimate: { unit: "USD_MICROCENTS", amount: 1_000_000 },
  });

  if (preflight.isSuccess) {
    const decision = preflight.getBodyAttribute("decision");
    if (decision === "DENY") {
      return NextResponse.json(
        { error: "budget_exceeded", message: "Insufficient budget." },
        { status: 402 },
      );
    }
  }

  // Budget allows — proceed with the LLM call
  const { prompt } = await req.json();
  // ... your withCycles-wrapped LLM call here ...
}
```

## Server Actions with budget governance

Guard Next.js Server Actions with `withCycles`:

```typescript
// app/actions.ts
"use server";

import { cyclesClient } from "@/lib/cycles";
import { withCycles, BudgetExceededError } from "runcycles";

const INPUT_PRICE = 250;
const OUTPUT_PRICE = 1_000;

export async function summarize(text: string) {
  const run = withCycles(
    {
      client: cyclesClient,
      actionKind: "llm.completion",
      actionName: "gpt-4o",
      estimate: () => Math.ceil(text.length / 4) * INPUT_PRICE + 512 * OUTPUT_PRICE,
      actual: (r: { usage: { prompt_tokens: number; completion_tokens: number } }) =>
        r.usage.prompt_tokens * INPUT_PRICE + r.usage.completion_tokens * OUTPUT_PRICE,
    },
    async () => {
      return callLLM(`Summarize: ${text}`, 512);
    },
  );

  try {
    const result = await run();
    return { content: result.content };
  } catch (err) {
    if (err instanceof BudgetExceededError) {
      return { error: "Budget exhausted. Try again later." };
    }
    throw err;
  }
}
```

## Per-tenant isolation

Extract the tenant from request headers or auth context:

```typescript
// lib/tenant.ts
import { headers } from "next/headers";

export async function getTenant(): Promise<string> {
  const headerList = await headers();
  return headerList.get("x-tenant-id") ?? "acme";
}
```

Use it in API routes to scope budget per tenant:

```typescript
// app/api/chat/route.ts
import { getTenant } from "@/lib/tenant";

export async function POST(req: Request) {
  const tenant = await getTenant();

  const generate = withCycles(
    {
      client: cyclesClient,
      actionKind: "llm.completion",
      actionName: "gpt-4o",
      tenant,
      estimate: () => 2_000_000,
      actual: (r: any) => r.usage.prompt_tokens * 250 + r.usage.completion_tokens * 1000,
    },
    async () => { /* LLM call */ },
  );

  // ...
}
```

`tenant` (and the other subject fields: `workspace`, `app`, `workflow`, `agent`, `toolset`) is a top-level `withCycles` option that overrides the client config default. Since 0.3.0 it also accepts a callable — `tenant: (...args) => string` — resolved per call from the wrapped function's arguments.

## Client-side error handling

Handle budget errors in React components:

```typescript
// components/chat.tsx
"use client";

import { useState } from "react";

export function Chat() {
  const [error, setError] = useState<string | null>(null);

  async function handleSubmit(prompt: string) {
    const res = await fetch("/api/chat", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ prompt }),
    });

    if (res.status === 402) {
      setError("Your budget has been exhausted. Please contact support.");
      return;
    }

    const data = await res.json();
    // handle response...
  }

  if (error) return <div className="error">{error}</div>;

  return <form onSubmit={(e) => { /* ... */ }}>{ /* ... */ }</form>;
}
```

## Key points

- **Singleton client in `lib/cycles.ts`.** Share one `CyclesClient` across all routes and server actions.
- **`withCycles` for API routes and server actions.** Wraps LLM calls with automatic reserve → execute → commit.
- **Route-handler preflight.** Do budget checks at the start of API routes or server actions; do not use the Node client from Edge middleware.
- **Per-tenant with headers.** Extract tenant from `x-tenant-id` header for multi-tenant budget isolation.
- **402 is an app-layer choice, not a Cycles convention.** Cycles signals budget exhaustion at the protocol layer (`DENY` / `BudgetExceededError`); returning 402 to the browser is one valid translation.
- **Use `runtime = "nodejs"`.** Required for `AsyncLocalStorage` support used by the Cycles client context.

## Next steps

- [Integrating with Vercel AI SDK](/how-to/integrating-cycles-with-vercel-ai-sdk) — streaming patterns with Vercel AI SDK
- [Handling Streaming Responses](/how-to/handling-streaming-responses-with-cycles) — streaming patterns in detail
- [Error Handling in TypeScript](/how-to/error-handling-patterns-in-typescript) — handling budget errors
- [Cost Estimation Cheat Sheet](/how-to/cost-estimation-cheat-sheet) — pricing reference for estimation
- [Production Operations Guide](/how-to/production-operations-guide) — running Cycles in production



# Integrating Cycles with Ollama

This guide shows how to guard [Ollama](https://ollama.com/) local LLM calls with Cycles budget reservations.

Budget control matters for local LLMs even though there are no per-token API charges. GPU time is a finite resource — shared inference servers have capacity limits, local GPUs have electricity and opportunity costs, and teams running models on shared infrastructure need visibility into who is consuming what. Cycles gives you the same reserve-execute-commit lifecycle for local models as you get for cloud APIs.

## Prerequisites

```bash
pip install runcycles ollama
```

Set environment variables:

```bash
export CYCLES_BASE_URL="http://localhost:7878"
export CYCLES_API_KEY="your-api-key"   # create via Admin Server — see note below
export CYCLES_TENANT="acme"
```

Make sure Ollama is running (`ollama serve`) and you have pulled a model:

```bash
ollama pull llama3.1
```

> **Need an API key?** Create one via the Admin Server — see [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack#step-3-create-an-api-key) or [API Key Management](/how-to/api-key-management-in-cycles).

::: tip 60-Second Quick Start
```python
import ollama
from runcycles import CyclesClient, CyclesConfig, cycles, set_default_client

set_default_client(CyclesClient(CyclesConfig.from_env()))

@cycles(estimate=500_000, action_kind="llm.completion", action_name="llama3.1")
def ask(prompt: str) -> str:
    response = ollama.chat(
        model="llama3.1",
        messages=[{"role": "user", "content": prompt}],
    )
    return response["message"]["content"]

print(ask("What is budget authority?"))
```
Every call is now budget-guarded. If the budget is exhausted, `BudgetExceededError` is raised _before_ Ollama is called. Read on for GPU-time cost estimation and multi-runner patterns.
:::

## Cost estimation for local models

Cloud APIs charge per token. Local models consume GPU time instead. A common approach is to assign a cost in microcents per GPU-second, then estimate based on expected inference duration:

```python
import time
import ollama
from runcycles import (
    CyclesClient, CyclesConfig, CyclesMetrics,
    cycles, get_cycles_context, set_default_client,
)

set_default_client(CyclesClient(CyclesConfig.from_env()))

# Cost in microcents per GPU-second (adjust for your hardware)
GPU_COST_PER_SECOND = 10_000  # e.g., $0.10/sec on an A100

@cycles(
    estimate=lambda prompt, **kw: GPU_COST_PER_SECOND * 30,  # assume 30s max
    actual=lambda result: result["cost"],
    action_kind="llm.completion",
    action_name="llama3.1",
    unit="USD_MICROCENTS",
    ttl_ms=120_000,
)
def chat(prompt: str) -> dict:
    start = time.monotonic()

    response = ollama.chat(
        model="llama3.1",
        messages=[{"role": "user", "content": prompt}],
    )

    elapsed = time.monotonic() - start
    cost = int(elapsed * GPU_COST_PER_SECOND)

    ctx = get_cycles_context()
    if ctx:
        ctx.metrics = CyclesMetrics(
            tokens_input=response.get("prompt_eval_count", 0),
            tokens_output=response.get("eval_count", 0),
            latency_ms=int(elapsed * 1000),
            model_version="llama3.1",
            custom={"gpu_seconds": round(elapsed, 2)},
        )

    return {
        "content": response["message"]["content"],
        "cost": cost,
    }
```

You can also budget purely by token count if you prefer — just assign a microcent value per token that reflects your infrastructure cost.

## Works with any Ollama-compatible runner

The `@cycles` pattern is not specific to the Ollama daemon. Any OpenAI-compatible local inference server works the same way. Simply swap the client:

```python
from openai import OpenAI
from runcycles import cycles

# vLLM, text-generation-inference, or any OpenAI-compatible server
local_client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")

@cycles(estimate=500_000, action_kind="llm.completion", action_name="llama3.1")
def ask_vllm(prompt: str) -> str:
    response = local_client.chat.completions.create(
        model="llama3.1",
        messages=[{"role": "user", "content": prompt}],
    )
    return response.choices[0].message.content
```

This works with [vLLM](https://docs.vllm.ai/), [text-generation-inference](https://huggingface.co/docs/text-generation-inference/), [LocalAI](https://localai.io/), or any server exposing an OpenAI-compatible endpoint.

## TypeScript example

Using the [ollama npm package](https://www.npmjs.com/package/ollama):

```typescript
import { Ollama } from "ollama";
import { CyclesClient, CyclesConfig, withCycles } from "runcycles";

const client = new CyclesClient(CyclesConfig.fromEnv());
const ollama = new Ollama();

const ask = withCycles(
  {
    client,
    estimate: 500_000,
    actionKind: "llm.completion",
    actionName: "llama3.1",
  },
  async (prompt: string): Promise<string> => {
    const response = await ollama.chat({
      model: "llama3.1",
      messages: [{ role: "user", content: prompt }],
    });
    return response.message.content;
  }
);

console.log(await ask("What is budget authority?"));
```

## Error handling

When the budget is insufficient, `BudgetExceededError` is raised **before** Ollama is called:

```python
from runcycles import BudgetExceededError

try:
    result = chat("Explain transformer architectures in detail")
except BudgetExceededError:
    result = {"content": "GPU budget exhausted — try again later.", "cost": 0}
```

For shared GPU infrastructure, a mandatory wrapper can stop a tenant's protected inference calls once its configured compute budget is unavailable. Scheduler fairness, GPU isolation, and calls that bypass the wrapper remain infrastructure concerns.

See [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) for patterns like queueing, model downgrade, and caching.

## Key points

- **Local does not mean free.** GPU time, electricity, and shared capacity all have real costs worth tracking.
- **GPU-time estimation.** Estimate cost by expected inference duration, then commit the actual GPU-seconds consumed.
- **Runner-agnostic.** The same `@cycles` pattern works with Ollama, vLLM, text-generation-inference, and any OpenAI-compatible server.
- **The model never runs on DENY.** If the budget is exhausted, no GPU time is consumed.

## Next steps

- [Error Handling Patterns in Python](/how-to/error-handling-patterns-in-python) — handling budget errors in Python
- [Testing with Cycles](/how-to/testing-with-cycles) — testing budget-guarded code
- [Production Operations Guide](/how-to/production-operations-guide) — running Cycles in production
- [Integrating with OpenAI](/how-to/integrating-cycles-with-openai) — for cloud OpenAI models
- [Integrating with Anthropic](/how-to/integrating-cycles-with-anthropic) — for Anthropic models



# OpenAI Agents SDK Budget Control (Python)

[![PyPI](https://img.shields.io/pypi/v/runcycles-openai-agents)](https://pypi.org/project/runcycles-openai-agents/)
[![PyPI downloads](https://img.shields.io/pypi/dm/runcycles-openai-agents?label=downloads&color=555&style=flat-square)](https://pypi.org/project/runcycles-openai-agents/)

This guide shows how to add budget governance to [OpenAI Agents SDK](https://github.com/openai/openai-agents-python) workflows using the [`runcycles-openai-agents`](https://pypi.org/project/runcycles-openai-agents/) plugin. The plugin hooks into the SDK's native `RunHooks` interface to automatically reserve, commit, and release budget for LLM calls and tool invocations, and attempts to record agent handoffs as zero-amount direct-debit events — with no per-function decoration required.

## Prerequisites

```bash
pip install runcycles-openai-agents
```

Set environment variables:

```bash
export OPENAI_API_KEY="sk-..."         # required by the OpenAI Agents SDK
export CYCLES_BASE_URL="http://localhost:7878"
export CYCLES_API_KEY="cyc_live_..."    # create via Admin Server — see note below
```

> **Prefer not to use environment variables?** All settings can be loaded programmatically from any secret manager, vault, or encrypted config file:
>
> ```python
> from runcycles import CyclesConfig, AsyncCyclesClient
> from runcycles_openai_agents import CyclesRunHooks
>
> config = CyclesConfig(
>     base_url=load_from_vault("cycles_base_url"),
>     api_key=load_from_vault("cycles_api_key"),
> )
> hooks = CyclesRunHooks(client=AsyncCyclesClient(config), tenant="acme")
> ```
>
> See [Python Client Configuration](/configuration/python-client-configuration-reference) for all options.

> **Need a Cycles server or API key?** See [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack) or [API Key Management](/how-to/api-key-management-in-cycles). For tenant and budget setup, see [Tenant Management](/how-to/tenant-creation-and-management-in-cycles) and [Budget Allocation](/how-to/budget-allocation-and-management-in-cycles).

::: tip 60-Second Quick Start
```python
from agents import Agent
from runcycles_openai_agents import CyclesRunHooks

hooks = CyclesRunHooks(tenant="acme")
agent = Agent(name="helper", instructions="You are a helpful assistant.")
result = await hooks.run(agent, input="What is budget authority?")
print(result.final_output)
```
That's it — every LLM call and tool invocation in the agent run is now budget-guarded. If the budget is exhausted, `BudgetExceededError` is raised _before_ the call is made. Read on for production patterns with tool estimate mapping and pre-run guardrails.
:::

## How it works

The plugin implements the SDK's `RunHooks` interface. Every hook in the agent lifecycle maps to a Cycles API call:

| Hook | Cycles API Call | Blocking | Detail |
|------|----------------|----------|--------|
| `on_tool_start` | `create_reservation` (tool estimate) | Raises on DENY | Budget reserved based on tool estimate map |
| `on_tool_end` | `commit_reservation` | No | Commits at the tool's reserved estimate |
| `on_llm_start` | `create_reservation` (LLM estimate) | Raises on DENY | Budget reserved before each LLM call |
| `on_llm_end` | `commit_reservation` | No | Commits at the reserved estimate — or at the actual token count when `llm_unit=Unit.TOKENS`. Token counts from `response.usage` are always recorded in `CyclesMetrics` |
| `on_handoff` | `create_event` (audit trail) | No | Attempts a zero-amount `RISK_POINTS` event; failures are logged and do not stop the handoff |

Reservations include automatic heartbeat. Extensions stop after `heartbeat_max_age_ms` (10 minutes by default) or the optional `heartbeat_max_extensions` cap, so raise those limits for longer operations. Heartbeat is skipped when `ttl_ms` is below 2000, since the minimum 1-second extend interval would exceed the TTL window.

## Tool estimate mapping

Assign per-call estimates to tools. Higher-estimate tools (send_email, deploy) consume budget faster. Zero-estimate tools skip the Cycles API entirely:

```python
from runcycles import Unit
from runcycles_openai_agents import CyclesRunHooks, ToolEstimateMap, ToolEstimateConfig

hooks = CyclesRunHooks(
    tenant="acme",
    tool_estimates=ToolEstimateMap(
        mapping={
            "send_email": 50,                       # 50 RISK_POINTS (default unit)
            "update_crm": ToolEstimateConfig(
                estimate=10,
                action_kind="tool.crm.update",
                unit=Unit.RISK_POINTS,              # explicit unit
            ),
            "search_knowledge": 0,                  # zero estimate — no reservation
        },
        default_estimate=1,                         # unmapped tools: 1 RISK_POINT
        default_unit=Unit.RISK_POINTS,              # unit for int shorthand values
    ),
)
```

Or use a simple dict:

```python
hooks = CyclesRunHooks(
    tenant="acme",
    tool_estimates={"send_email": 50, "search": 0},  # default unit: RISK_POINTS
)
```

## Pre-run guardrail

`cycles_budget_guardrail` returns an `InputGuardrail` that calls `/v1/decide` before the agent starts. If the tenant is suspended or budget is exhausted, the guardrail trips and the agent never runs — zero tokens consumed:

```python
from agents import Agent
from runcycles_openai_agents import cycles_budget_guardrail

guardrail = cycles_budget_guardrail(
    tenant="acme",
    estimate=5_000_000,       # expected total run estimate
    fail_open=True,           # allow if Cycles server is down
)

agent = Agent(
    name="support-bot",
    input_guardrails=[guardrail],
)
```

## Error handling

When budget is denied, the hooks raise `BudgetExceededError` — the agent stops and no further tokens are consumed. Use `hooks.run()` so failures and cancellations also release reservations that are still pending:

```python
from runcycles import BudgetExceededError

try:
    result = await hooks.run(agent, input="...")
except BudgetExceededError as e:
    print(f"Budget denied: {e}")
```

`hooks.run()` wraps `Runner.run()` and performs run-scoped cleanup when the run raises or is cancelled. Streaming runs get the same boundary through `hooks.run_streamed()`:

```python
streamed = hooks.run_streamed(agent, input="...")
async for event in streamed.stream_events():
    handle(event)
```

The OpenAI Agents SDK does not expose a general `RunHooks.on_error` callback. If you call bare `Runner.run(..., hooks=hooks)` or `Runner.run_streamed(..., hooks=hooks)`, automatic exception/cancellation cleanup cannot run. Prefer the wrappers, especially for concurrent runs. In a single-run bare-Runner error path, call `release_pending()`; it raises rather than guessing when multiple runs have pending reservations.

See [Error Handling Patterns in Python](/how-to/error-handling-patterns-in-python) for more patterns.

## Fail-open / fail-closed

`CyclesRunHooks` is fail-closed by default (`fail_open=False`): a transport failure while creating a reservation raises and prevents the governed call. Opt into availability-first behavior explicitly:

```python
hooks = CyclesRunHooks(tenant="acme", fail_open=True)
```

The separate `cycles_budget_guardrail()` helper defaults to `fail_open=True`; pass `fail_open=False` there if the pre-run decision must also fail closed.

A successful non-DENY response that omits `reservation_id` is currently logged and allowed to proceed without a hold. Alert on the `cycles: no reservation_id` warning if malformed upstream responses must not pass unnoticed.

## Configuration reference

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `client` | `AsyncCyclesClient` | `None` | Explicit client (or auto-created from config/env) |
| `config` | `CyclesConfig` | `None` | Creates client if no client given |
| `tenant` | `str` | `None` | Subject.tenant |
| `workspace` | `str` | `None` | Subject.workspace |
| `app` | `str` | `None` | Subject.app |
| `workflow` | `str` | `None` | Subject.workflow |
| `agent` | `str` | `None` | Subject.agent (overridden by actual agent name) |
| `toolset` | `str` | `None` | Subject.toolset |
| `tool_estimates` | `dict` or `ToolEstimateMap` | `{}` | Tool name → per-call estimate (default unit: RISK_POINTS) |
| `default_tool_estimate` | `int` | `1` | Estimate for unmapped tools |
| `llm_estimate` | `int` | `500_000` | Per-LLM-call estimate (~$0.005 in USD_MICROCENTS) |
| `llm_unit` | `Unit` | `USD_MICROCENTS` | Unit for LLM reservations |
| `fail_open` | `bool` | `False` | Allow LLM/tool execution if reservation creation fails |
| `ttl_ms` | `int` | `60_000` | Reservation TTL (heartbeat extends at half-interval) |
| `heartbeat_max_age_ms` | `int` | `600_000` | Maximum age of a pending operation for heartbeat extension |
| `heartbeat_max_extensions` | `int` or `None` | `None` | Optional cap on heartbeat extensions per reservation |
| `commit_max_attempts` | `int` | `2` | Attempts for transport, 429, and 5xx commit failures |
| `overage_policy` | `CommitOveragePolicy` | `ALLOW_IF_AVAILABLE` | Overage policy for commits |
| `dry_run` | `bool` | `False` | Shadow mode — no budget consumed |

## Comparison with manual integration

If you're already using the `@cycles` decorator from the [Python client](/quickstart/getting-started-with-the-python-client), the plugin automates the same reserve → commit → release pattern at the agent framework level:

| Concern | `@cycles` decorator | `CyclesRunHooks` plugin |
|---------|---------------------|-------------------------|
| Reserve before LLM call | Your code (per function) | Automatic via `on_llm_start` |
| Reserve before tool call | Your code (per function) | Automatic via `on_tool_start` |
| Commit after completion | Your code (per function) | Automatic via `on_llm_end` / `on_tool_end` |
| Release on run error or cancellation | Your code | Automatic with `hooks.run()` / `hooks.run_streamed()`; `release_pending()` for a single bare-Runner error path |
| Tool estimate policies | Not applicable | `ToolEstimateMap` with per-tool estimates |
| Pre-run guardrail | Not applicable | `cycles_budget_guardrail` |
| Agent handoff tracking | Not applicable | Best-effort audit events via `on_handoff` |
| Heartbeat for long tools | Not applicable | Automatic TTL extension |

The plugin is the recommended approach for OpenAI Agents SDK users. It requires no per-function decoration; use its run wrappers so hook governance and run-scoped cleanup stay paired. Handoff audit events are best effort and log failures without blocking the handoff.

## Examples

See the [`examples/`](https://github.com/runcycles/cycles-openai-agents/tree/main/examples) directory for runnable integration examples:

| Example | Description |
|---------|-------------|
| [`basic_budget.py`](https://github.com/runcycles/cycles-openai-agents/blob/main/examples/basic_budget.py) | LLM token budget enforcement |
| [`tool_governance.py`](https://github.com/runcycles/cycles-openai-agents/blob/main/examples/tool_governance.py) | Tool estimate mapping — higher-estimate tools consume more |
| [`multi_agent.py`](https://github.com/runcycles/cycles-openai-agents/blob/main/examples/multi_agent.py) | Multi-agent handoff with shared budget |

## Next steps

- [Error Handling Patterns in Python](/how-to/error-handling-patterns-in-python) — handling budget errors
- [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) — strategies for graceful degradation
- [Testing with Cycles](/how-to/testing-with-cycles) — testing budget-guarded code
- [Production Operations Guide](/how-to/production-operations-guide) — running Cycles in production
- [Cost Estimation Cheat Sheet](/how-to/cost-estimation-cheat-sheet) — pricing reference for estimation
- [runcycles-openai-agents on PyPI](https://pypi.org/project/runcycles-openai-agents/) — package page
- [Source on GitHub](https://github.com/runcycles/cycles-openai-agents) — full source code and examples

## Related concepts

- [Audit trail as a runtime-authority byproduct](/blog/runtime-authority-byproducts-audit-trail-and-attribution-by-default)
- [Graceful degradation patterns](/blog/when-budget-runs-out-graceful-degradation-patterns-for-ai-agents)
- [Multi-agent coordination failure: structural prevention](/blog/multi-agent-coordination-failure-structural-prevention)



# Integrating Cycles with OpenAI (TypeScript)

This guide shows how to guard OpenAI API calls with Cycles budget reservations in TypeScript, including streaming support and caps-aware completions.

For the Python version, see [Integrating with OpenAI (Python)](/how-to/integrating-cycles-with-openai).

## Prerequisites

- A running Cycles stack with a tenant, API key, and budget ([Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack))
- Node.js 20+

## Installation

```bash
npm install runcycles openai
```

```bash
export CYCLES_BASE_URL="http://localhost:7878"
export CYCLES_API_KEY="cyc_live_..."
export OPENAI_API_KEY="sk-..."
```

::: tip 60-Second Quick Start
```typescript
import OpenAI from "openai";
import { CyclesClient, CyclesConfig, withCycles } from "runcycles";

const cycles = new CyclesClient(CyclesConfig.fromEnv());
const openai = new OpenAI();

const ask = withCycles(
  {
    client: cycles,
    actionKind: "llm.completion",
    actionName: "gpt-5.6-luna",
    estimate: () => 1_000_000,
    actual: (r: OpenAI.ChatCompletion) =>
      (r.usage?.prompt_tokens ?? 0) * 100 + (r.usage?.completion_tokens ?? 0) * 600,
  },
  async (prompt: string) => {
    return openai.chat.completions.create({
      model: "gpt-5.6-luna",
      messages: [{ role: "user", content: prompt }],
    });
  },
);

const response = await ask("What is budget authority?");
console.log(response.choices[0].message.content);
```
Budget is reserved before the call and committed with actual token cost after. If budget is exhausted, `BudgetExceededError` is thrown _before_ the OpenAI call is made.
:::

## Non-streaming calls with withCycles

Use the `withCycles` higher-order function to wrap OpenAI calls with automatic reserve → execute → commit:

```typescript
import OpenAI from "openai";
import {
  CyclesClient, CyclesConfig, withCycles,
  setDefaultClient, getCyclesContext, BudgetExceededError,
} from "runcycles";

const cyclesClient = new CyclesClient(CyclesConfig.fromEnv());
setDefaultClient(cyclesClient);

const openai = new OpenAI();

// GPT-5.6 Luna standard pricing (microcents per token)
const MODEL = "gpt-5.6-luna";
const INPUT_PRICE = 100;   // $1.00 / 1M uncached input tokens
const OUTPUT_PRICE = 600;  // $6.00 / 1M output tokens
const DEFAULT_MAX_TOKENS = 1024;

const chatCompletion = withCycles(
  {
    client: cyclesClient,
    actionKind: "llm.completion",
    actionName: MODEL,
    estimate: (prompt: string) => {
      const inputTokens = Math.ceil(prompt.length / 4);
      return inputTokens * INPUT_PRICE + DEFAULT_MAX_TOKENS * OUTPUT_PRICE;
    },
    actual: (response: OpenAI.ChatCompletion) => {
      return (response.usage?.prompt_tokens ?? 0) * INPUT_PRICE
        + (response.usage?.completion_tokens ?? 0) * OUTPUT_PRICE;
    },
  },
  async (prompt: string) => {
    const ctx = getCyclesContext();

    // Respect budget caps
    let maxTokens = DEFAULT_MAX_TOKENS;
    if (ctx?.caps?.maxTokens) {
      maxTokens = Math.min(maxTokens, ctx.caps.maxTokens);
    }

    const response = await openai.chat.completions.create({
      model: MODEL,
      max_completion_tokens: maxTokens,
      messages: [{ role: "user", content: prompt }],
    });

    // Report metrics for observability
    if (ctx) {
      ctx.metrics = {
        tokensInput: response.usage?.prompt_tokens,
        tokensOutput: response.usage?.completion_tokens,
        modelVersion: response.model,
      };
    }

    return response;
  },
);

try {
  const response = await chatCompletion("Explain budget governance.");
  console.log(response.choices[0].message.content);
} catch (err) {
  if (err instanceof BudgetExceededError) {
    console.log("Budget exhausted.");
  } else {
    throw err;
  }
}
```

## Streaming with reserveForStream

For streaming responses, use `reserveForStream` to manage the reservation lifecycle:

```typescript
import OpenAI from "openai";
import {
  CyclesClient, CyclesConfig, reserveForStream, BudgetExceededError,
} from "runcycles";

const cyclesClient = new CyclesClient(CyclesConfig.fromEnv());
const openai = new OpenAI();

const MODEL = "gpt-5.6-luna";
const INPUT_PRICE = 100;
const OUTPUT_PRICE = 600;

async function streamWithBudget(prompt: string) {
  const estimatedInputTokens = Math.ceil(prompt.length / 4);
  const estimate = estimatedInputTokens * INPUT_PRICE + 1024 * OUTPUT_PRICE;

  // 1. Reserve budget
  const handle = await reserveForStream({
    client: cyclesClient,
    estimate,
    unit: "USD_MICROCENTS",
    actionKind: "llm.completion",
    actionName: MODEL,
  });

  try {
    // Respect budget caps
    let maxTokens = 1024;
    if (handle.caps?.maxTokens) {
      maxTokens = Math.min(maxTokens, handle.caps.maxTokens);
    }

    // 2. Stream the response
    const stream = await openai.chat.completions.create({
      model: MODEL,
      max_completion_tokens: maxTokens,
      messages: [{ role: "user", content: prompt }],
      stream: true,
      stream_options: { include_usage: true },
    });

    let promptTokens = 0;
    let completionTokens = 0;

    for await (const chunk of stream) {
      const text = chunk.choices[0]?.delta?.content;
      if (text) process.stdout.write(text);

      if (chunk.usage) {
        promptTokens = chunk.usage.prompt_tokens ?? 0;
        completionTokens = chunk.usage.completion_tokens ?? 0;
      }
    }

    // 3. Commit actual usage
    const actualCost = promptTokens * INPUT_PRICE + completionTokens * OUTPUT_PRICE;
    await handle.commit(actualCost, {
      tokensInput: promptTokens,
      tokensOutput: completionTokens,
      modelVersion: MODEL,
    });
  } catch (err) {
    await handle.release("stream_error");
    throw err;
  }
}
```

::: info max_tokens is deprecated
The snippets use `max_completion_tokens`: OpenAI deprecated `max_tokens` in its favor, and reasoning models (o-series) reject `max_tokens` outright.
:::

## Pricing reference

Adjust these constants for the model you use:

| Model | Input (microcents/token) | Output (microcents/token) |
|-------|--------------------------|---------------------------|
| gpt-5.6-sol | 500 | 3,000 |
| gpt-5.6-terra | 250 | 1,500 |
| gpt-5.6-luna | 100 | 600 |
| gpt-4o | 250 | 1,000 |
| gpt-4o-mini | 15 | 60 |
| gpt-4.1 | 200 | 800 |
| gpt-4.1-mini | 40 | 160 |
| gpt-4.1-nano | 10 | 40 |
| o3 | 200 | 800 |
| o4-mini | 110 | 440 |

The snippets use standard uncached GPT-5.6 Luna pricing for requests with at most 272,000 input tokens. GPT-5.6 cache reads are cheaper, explicit cache writes cost 1.25 times the uncached input rate, and longer requests use higher rates for the full request. If you use those features, compute actual cost from the provider's detailed usage fields. OpenAI recommends the Responses API for reasoning, tool-calling, and multi-turn workflows; Chat Completions remains supported for these single-turn examples. See the [GPT-5.6 model guide](https://developers.openai.com/api/docs/guides/latest-model) and [GPT-5.6 Luna model page](https://developers.openai.com/api/docs/models/gpt-5.6-luna).

See [Cost Estimation Cheat Sheet](/how-to/cost-estimation-cheat-sheet) for the full pricing reference.

## Key points

- **`withCycles` for non-streaming.** Wraps a single OpenAI call with automatic reserve → execute → commit.
- **`reserveForStream` for streaming.** Manages the reservation lifecycle with automatic heartbeat during the stream.
- **Use `stream_options: { include_usage: true }`.** Required to get token counts from OpenAI streaming responses.
- **Token fields:** `usage.prompt_tokens` / `usage.completion_tokens` (OpenAI naming).
- **Respect caps.** Check `ctx.caps?.maxTokens` or `handle.caps?.maxTokens` to honor budget authority limits.

## Full example

See [`examples/openai-sdk/`](https://github.com/runcycles/cycles-client-typescript/tree/main/examples/openai-sdk) for a complete, runnable example.

## Next steps

- [Integrating with OpenAI (Python)](/how-to/integrating-cycles-with-openai) — Python version of this guide
- [Handling Streaming Responses](/how-to/handling-streaming-responses-with-cycles) — streaming patterns in detail
- [Cost Estimation Cheat Sheet](/how-to/cost-estimation-cheat-sheet) — pricing reference for estimation
- [Error Handling in TypeScript](/how-to/error-handling-patterns-in-typescript) — handling budget errors
- [Production Operations Guide](/how-to/production-operations-guide) — running Cycles in production



# Integrating Cycles with OpenAI

This guide shows how to guard OpenAI API calls with Cycles budget reservations so that every chat completion is cost-controlled, caps-aware, and observable.

::: tip Using the OpenAI Agents SDK?
If you're building multi-agent workflows with the [OpenAI Agents SDK](https://github.com/openai/openai-agents-python), see [Integrating Cycles with OpenAI Agents SDK](/how-to/integrating-cycles-with-openai-agents) instead — it covers the entire agent run automatically with no per-function decoration.
:::

## Prerequisites

```bash
pip install runcycles openai
```

Set environment variables:

```bash
export CYCLES_BASE_URL="http://localhost:7878"
export CYCLES_API_KEY="your-api-key"   # create via Admin Server — see note below
export CYCLES_TENANT="acme"
export OPENAI_API_KEY="sk-..."
```

> **Prefer not to use environment variables?** All settings can be loaded programmatically from any secret manager, vault, or encrypted config file:
>
> ```python
> from runcycles import CyclesConfig, CyclesClient, set_default_client
>
> config = CyclesConfig(
>     base_url=load_from_vault("cycles_base_url"),
>     api_key=load_from_vault("cycles_api_key"),
>     tenant=load_from_vault("cycles_tenant"),
> )
> set_default_client(CyclesClient(config))
> ```
>
> See [Python Client Configuration](/configuration/python-client-configuration-reference) for all options.

> **Need an API key?** Create one via the Admin Server — see [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack#step-3-create-an-api-key) or [API Key Management](/how-to/api-key-management-in-cycles).

::: tip 60-Second Quick Start
```python
from openai import OpenAI
from runcycles import CyclesClient, CyclesConfig, cycles, set_default_client

set_default_client(CyclesClient(CyclesConfig.from_env()))

@cycles(estimate=1_000_000, action_kind="llm.completion", action_name="gpt-5.6-luna")
def ask(prompt: str) -> str:
    return OpenAI().chat.completions.create(
        model="gpt-5.6-luna",
        messages=[{"role": "user", "content": prompt}],
    ).choices[0].message.content

print(ask("What is budget authority?"))
```
That's it — every call is now budget-guarded. If the budget is exhausted, `BudgetExceededError` is raised _before_ the OpenAI call is made.

> **Note:** This quick start commits the estimate as actual spend. For accurate cost tracking, add an `actual` callback — see [Basic pattern](#basic-pattern) below.
:::

## Basic pattern

Use the `@cycles` decorator to wrap an OpenAI call with automatic reserve → execute → commit:

```python
from openai import OpenAI
from runcycles import (
    CyclesClient, CyclesConfig, CyclesMetrics,
    cycles, get_cycles_context, set_default_client,
)

# Set up clients
config = CyclesConfig.from_env()
set_default_client(CyclesClient(config))
openai_client = OpenAI()

# GPT-5.6 Luna standard pricing in USD microcents
# (1 USD = 100_000_000 microcents)
MODEL = "gpt-5.6-luna"
PRICE_PER_INPUT_TOKEN = 100      # $1.00 / 1M uncached input tokens
PRICE_PER_OUTPUT_TOKEN = 600     # $6.00 / 1M output tokens

@cycles(
    estimate=lambda prompt, **kw: len(prompt.split()) * 2 * PRICE_PER_INPUT_TOKEN
        + kw.get("max_tokens", 1024) * PRICE_PER_OUTPUT_TOKEN,
    actual=lambda result: (
        result["usage"]["prompt_tokens"] * PRICE_PER_INPUT_TOKEN
        + result["usage"]["completion_tokens"] * PRICE_PER_OUTPUT_TOKEN
    ),
    action_kind="llm.completion",
    action_name=MODEL,
    unit="USD_MICROCENTS",
    ttl_ms=60_000,
)
def chat_completion(prompt: str, max_tokens: int = 1024) -> dict:
    ctx = get_cycles_context()

    # Respect caps from the budget authority
    if ctx and ctx.has_caps() and ctx.caps.max_tokens:
        max_tokens = min(max_tokens, ctx.caps.max_tokens)

    response = openai_client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": prompt}],
        max_completion_tokens=max_tokens,
    )

    # Report metrics
    if ctx:
        ctx.metrics = CyclesMetrics(
            tokens_input=response.usage.prompt_tokens,
            tokens_output=response.usage.completion_tokens,
            model_version=response.model,
        )

    return {
        "content": response.choices[0].message.content,
        "usage": {
            "prompt_tokens": response.usage.prompt_tokens,
            "completion_tokens": response.usage.completion_tokens,
        },
    }
```

## Cost estimation strategies

The estimate function runs **before** the API call. The more accurate it is, the less budget you hold unnecessarily:

| Strategy | Accuracy | Example |
|----------|----------|---------|
| Constant | Low | `estimate=500_000` |
| Token-proportional | Medium | `estimate=lambda p, **kw: kw.get("max_tokens", 1024) * PRICE_PER_OUTPUT_TOKEN` |
| Input + output | High | Count input tokens (or approximate from word count) plus max output tokens |

For production use, consider using `tiktoken` for accurate input token counts:

```python
import tiktoken

try:
    enc = tiktoken.encoding_for_model(MODEL)
except KeyError:
    # Keep this fallback aligned with the model's documented tokenizer.
    enc = tiktoken.get_encoding("o200k_base")

def estimate_cost(prompt: str, max_tokens: int = 1024) -> int:
    input_tokens = len(enc.encode(prompt))
    return (
        input_tokens * PRICE_PER_INPUT_TOKEN
        + max_tokens * PRICE_PER_OUTPUT_TOKEN
    )
```

The constants above use standard uncached pricing for requests with at most 272,000 input tokens. GPT-5.6 cache reads are cheaper, explicit cache writes cost 1.25 times the uncached input rate, and longer requests use higher rates for the full request. If you use those features, calculate `actual` from the provider's detailed usage fields rather than the simplified formula above. OpenAI recommends the Responses API for reasoning, tool-calling, and multi-turn workflows; Chat Completions remains supported for this single-turn example. See the [GPT-5.6 model guide](https://developers.openai.com/api/docs/guides/latest-model) and [GPT-5.6 Luna model page](https://developers.openai.com/api/docs/models/gpt-5.6-luna).

## Handling budget exhaustion

When the budget is insufficient, the `@cycles` decorator raises `BudgetExceededError` **without** calling OpenAI:

```python
from runcycles import BudgetExceededError

try:
    result = chat_completion("Summarize this document...")
except BudgetExceededError:
    # Degrade gracefully
    result = {"content": "Service temporarily unavailable.", "usage": {}}
```

See [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) for patterns like queueing, model downgrade, and caching.

## Respecting caps

When the decision is `ALLOW_WITH_CAPS`, the budget authority may limit token usage. Always check and respect caps inside your function:

```python
ctx = get_cycles_context()
if ctx and ctx.has_caps() and ctx.caps.max_tokens:
    max_tokens = min(max_tokens, ctx.caps.max_tokens)
```

This lets the budget authority throttle expensive requests without fully denying them.

## Reporting metrics

Metrics attached to the context are included in the commit and become available for observability:

```python
ctx.metrics = CyclesMetrics(
    tokens_input=response.usage.prompt_tokens,
    tokens_output=response.usage.completion_tokens,
    latency_ms=elapsed_ms,
    model_version=response.model,
)
```

## Key points

- **Estimate before, commit after.** The `estimate` function determines how much budget to reserve; the `actual` function computes the real cost from the response.
- **Caps are advisory.** The budget authority sets them; your code decides how to enforce them.
- **Metrics are optional but valuable.** They flow into Cycles for per-model, per-tenant cost visibility.
- **The function never executes on DENY.** OpenAI is never called if the budget is exhausted, saving both money and latency.

## Full example

See [`examples/openai_integration.py`](https://github.com/runcycles/cycles-client-python/blob/main/examples/openai_integration.py) for a complete, runnable script.

## Next steps

- [Integrating with OpenAI (TypeScript)](/how-to/integrating-cycles-with-openai-typescript) — TypeScript version of this guide
- [Integrating with OpenAI Agents SDK](/how-to/integrating-cycles-with-openai-agents) — budget governance for multi-agent workflows
- [Error Handling Patterns in Python](/how-to/error-handling-patterns-in-python) — handling budget errors in Python
- [Handling Streaming Responses](/how-to/handling-streaming-responses-with-cycles) — budget-managed streaming
- [Testing with Cycles](/how-to/testing-with-cycles) — testing budget-guarded code
- [Production Operations Guide](/how-to/production-operations-guide) — running Cycles in production
- [OpenAI example (TypeScript)](https://github.com/runcycles/cycles-client-typescript/tree/main/examples/openai-sdk) — runnable OpenAI SDK integration
- [OpenAI example (Python)](https://github.com/runcycles/cycles-client-python/blob/main/examples/openai_integration.py) — runnable OpenAI integration



# Integrating Cycles with OpenClaw

[![npm](https://img.shields.io/npm/v/@runcycles/openclaw-budget-guard)](https://www.npmjs.com/package/@runcycles/openclaw-budget-guard)
[![npm downloads](https://img.shields.io/npm/dt/@runcycles/openclaw-budget-guard?label=downloads&color=555&style=flat-square)](https://www.npmjs.com/package/@runcycles/openclaw-budget-guard)

This guide shows how to add budget enforcement to OpenClaw agents using the [`cycles-openclaw-budget-guard`](https://github.com/runcycles/cycles-openclaw-budget-guard) plugin. The plugin handles the full reserve → commit → release lifecycle for both model and tool calls automatically, with no custom code required.

## Choose your path

- **Cap OpenClaw agents in JS/TS** — install [`@runcycles/openclaw-budget-guard`](https://www.npmjs.com/package/@runcycles/openclaw-budget-guard) and follow this guide.
- **Use Cycles directly without OpenClaw** — start with the [TypeScript client](/quickstart/getting-started-with-the-typescript-client) for custom agent frameworks.
- **Evaluate Cycles for a multi-tenant agent SaaS** — start with the [evaluation guide](/how-to/evaluate-cycles-for-agent-saas) before reading this implementation page.
- **Not sure where Cycles fits?** [Send us your tool-call flow](/contact) and we'll map where `reserve` / `commit` should sit.

## Why budget enforcement?

AI agents make autonomous decisions — calling models, invoking tools, retrying on failure — with no human in the loop. Without runtime enforcement:

- **Runaway spend** — a single [runaway agent](/incidents/runaway-agents-tool-loops-and-budget-overruns-the-incidents-cycles-is-designed-to-prevent) can consume substantial budget quickly. Provider budgets, credits, and quotas use their own scopes and semantics; request-count limits do not account for variable call cost.
- **Uncontrolled side-effects** — an agent can send hundreds of emails, trigger deployments, or call dangerous APIs with nothing to stop it. Cost limits alone don't help — some actions are consequential regardless of price.
- **Noisy neighbors** — in multi-tenant or multi-user setups, one agent can consume the entire team budget, starving other users.
- **No session-level cost visibility** — when a session ends, you have no idea what it spent, which tools it called most, or whether it was cost-efficient.
- **Abrupt failure** — budget runs out and the agent crashes instead of adapting.

The plugin adds budget checks to model and tool calls that pass through OpenClaw's supported lifecycle hooks. With `modelFallbacks`, low-budget strategies, and `toolCallLimits` configured, it can downgrade matching models, disable expensive tools, add remaining-budget prompt hints, and cap per-tool invocation counts. Standard subject fields in `budgetScope` select enforceable tenant, workspace, app, workflow, agent, or toolset ledgers; `userId` and `sessionId` are attribution dimensions unless you map them to those fields. Session summaries report the plugin's estimated model and tool costs.

Beyond enforcement, the plugin actively protects you:

- **Burn rate anomaly detection** emits `cycles.budget.burn_rate_anomaly` when the current window exceeds the configured comparison threshold; your OTLP backend decides whether to alert or intervene
- **Predictive exhaustion warnings** estimate when budget will run out and emit `cycles.budget.exhaustion_forecast_ms` before it happens, so you can fund the budget or wind down gracefully
- **Automatic retry with backoff** on transient Cycles server errors (429/503/504) prevents spurious denials during load spikes
- **Reservation heartbeat** auto-extends long-running tool reservations so cost tracking doesn't silently break when a tool exceeds the default 60s TTL
- **Budget observability** via the built-in OTLP HTTP adapter (14 metrics when the current features are enabled, configured through `otlpMetricsEndpoint`) and opt-in session logs of the plugin's reserve, commit, deny, block, and release decisions
- **Unconfigured tool detection** reports which tools are using default cost estimates so you can tune `toolBaseCosts` after every session

The result is a mandatory budget boundary for the hook paths the plugin covers, plus configurable degradation and budget telemetry. External model and tool outcomes still belong in application or provider logs.

Install, configure 3 fields, done. No agent code changes required.

::: tip When to use this vs. the Cycles client directly
If you're building a custom agent framework, use the [Cycles TypeScript client](/how-to/using-the-cycles-client-programmatically) directly. If you're running OpenClaw, this plugin gives you the same enforcement with zero custom code — just configure and go.
:::

For background on why rate limits and provider caps aren't enough, see [Exposure: Why Rate Limits Leave Agents Unbounded](/concepts/exposure-why-rate-limits-leave-agents-unbounded) and [Cycles vs. Provider Spending Caps](/concepts/cycles-vs-provider-spending-caps).

## Quick start

Get budget enforcement running in under a minute:

```bash
openclaw plugins install @runcycles/openclaw-budget-guard
openclaw plugins enable openclaw-budget-guard
```

Add minimal configuration to your OpenClaw config file (typically `openclaw.json` or `openclaw.config.json`):

```json
{
  "plugins": {
    "entries": {
      "openclaw-budget-guard": {
        "config": {
          "cyclesBaseUrl": "http://localhost:7878",
          "cyclesApiKey": "cyc_live_...",
          "tenant": "my-org",
          "currency": "USD_MICROCENTS"
        }
      }
    }
  }
}
```

That's it — all model and tool calls are now budget-guarded. Read on for advanced features like model fallbacks, tool access control, and budget-aware prompt hints.

## Prerequisites

- A Cycles API key — create one via the Admin Server. See [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack#step-3-create-an-api-key) or [API Key Management](/how-to/api-key-management-in-cycles).

You also need:
- **OpenClaw** >= 0.1.0 with plugin support
- **Node.js** >= 20.0.0

## Install and enable

```bash
# Install the plugin
openclaw plugins install @runcycles/openclaw-budget-guard

# Enable it
openclaw plugins enable openclaw-budget-guard
```

Or install from a local checkout:

```bash
openclaw plugins install -l ./cycles-openclaw-budget-guard
openclaw plugins enable openclaw-budget-guard
```

## Minimal configuration

Add the plugin to your OpenClaw config file (typically `openclaw.config.json`). Three fields are required — everything else has sensible defaults:

```json
{
  "plugins": {
    "entries": {
      "openclaw-budget-guard": {
        "config": {
          "cyclesBaseUrl": "http://localhost:7878",
          "cyclesApiKey": "your-api-key",
          "tenant": "acme"
        }
      }
    }
  }
}
```

> **Important:** Budget exhaustion and reservation failures are enforced fail-closed by default. A failed budget-snapshot fetch is treated as healthy by default, but you can make that path fail closed with `failClosedOnSnapshotError: true`. See [Fail-open vs fail-closed](#fail-open-vs-fail-closed) for details.

## Understanding the cost model

Every model call and tool call reserves a fixed cost from the budget. The default currency is `USD_MICROCENTS` — 1 unit = $0.00000001 (10⁻⁸ dollars).

| Amount | USD |
|--------|-----|
| 100,000 | $0.001 |
| 1,000,000 | $0.01 |
| 10,000,000 | $0.10 |
| 100,000,000 | $1.00 |

**Example.** With a $5 budget (500,000,000 units) and `claude-opus` at 1,500,000/call, you can afford ~333 model calls. The `lowBudgetThreshold` (default 10,000,000 = $0.10) triggers model downgrade when budget is nearly exhausted.

**Setting tool costs.** Start with defaults (100,000/call). After your first session, check `sessionSummary.unconfiguredTools` for the list of tools that need explicit costs. External API tools (web search, code execution) typically cost 500K-1M. Lightweight tools (text formatting, math) cost 10K-50K.

## What the plugin does

The plugin hooks into five OpenClaw lifecycle events to enforce budget boundaries:

| Hook | What happens |
|------|-------------|
| `before_model_resolve` | Fetches balance, reserves budget for the model call, downgrades the model if budget is low, blocks if exhausted (via [model override workaround](#model-blocking-workaround-v0-7-3)). The reservation is held open for later commit (see [Model cost reconciliation](#model-cost-reconciliation-v0-5-0)). |
| `before_prompt_build` | Commits any pending model reservation from the previous turn at the reserved estimate. Injects a budget-awareness hint into the system prompt, including forecast projections and pool balances. |
| `before_tool_call` | Checks tool permissions (allowlist/blocklist), applies degradation strategies, creates a Cycles reservation. Optionally retries on denial. |
| `after_tool_call` | Commits the reservation with the estimated cost from `toolBaseCosts` (or the default). |
| `agent_end` | Releases orphaned reservations, builds a session summary with cost breakdown and forecasts, POSTs it to `analyticsWebhookUrl` if configured. |

Both model and tool calls follow the standard Cycles reserve → commit → release protocol. The plugin manages an in-memory map of active reservations so that every reservation is properly settled or released at `agent_end`.

## Budget levels and model downgrading

The plugin classifies budget into three levels:

| Level | Condition | Behavior |
|-------|-----------|----------|
| **healthy** | `remaining > lowBudgetThreshold` | Pass through — no changes |
| **low** | `exhaustedThreshold < remaining ≤ lowBudgetThreshold` | Apply low-budget strategies (model downgrade, token limits, tool restrictions) |
| **exhausted** | `remaining ≤ exhaustedThreshold` | Block execution (or warn, if `failClosed: false`) |

### Chained model fallbacks

Model fallbacks support both single values and ordered chains. When budget is low, the plugin iterates through candidates and selects the first one whose cost fits within the remaining budget:

```json
{
  "plugins": {
    "entries": {
      "openclaw-budget-guard": {
        "config": {
          "tenant": "acme",
          "modelFallbacks": {
            "anthropic/claude-opus-4-8": ["anthropic/claude-sonnet-4-6", "anthropic/claude-haiku-4-5-20251001"],
            "openai/gpt-4o": "openai/gpt-4o-mini"
          },
          "modelBaseCosts": {
            "anthropic/claude-opus-4-8": 500000,
            "anthropic/claude-sonnet-4-6": 300000,
            "anthropic/claude-haiku-4-5-20251001": 100000,
            "openai/gpt-4o": 1000000,
            "openai/gpt-4o-mini": 100000
          }
        }
      }
    }
  }
}
```

When the budget drops below `lowBudgetThreshold` (default: 10,000,000 units), any model request matching a key in `modelFallbacks` is transparently swapped to the cheapest affordable alternative.

## Tool cost estimation

Configure per-tool cost estimates via `toolBaseCosts`:

```json
{
  "plugins": {
    "entries": {
      "openclaw-budget-guard": {
        "config": {
          "tenant": "acme",
          "toolBaseCosts": {
            "web_search": 500000,
            "code_execution": 1000000,
            "file_read": 50000
          }
        }
      }
    }
  }
}
```

Any tool not listed defaults to 100,000 units.

::: warning Type safety note
Config values in `toolBaseCosts` and `modelBaseCosts` are validated at startup (v0.7.10+). Non-number values are rejected and negative values throw an error. If using OpenClaw's JSON config, ensure all cost values are numbers, not quoted strings.
:::

::: info Tuning tool costs
OpenClaw plugins are configured via JSON, which cannot carry JavaScript functions, so the plugin's `costEstimator` callback is **not available** through OpenClaw config. To refine estimates, iterate on `toolBaseCosts` using session data:

1. Set `enableEventLog: true` in your plugin config.
2. Run a few agent sessions.
3. Inspect the session summary in the logs — it includes per-tool cost breakdowns and an `unconfiguredTools` list of tools falling back to the default estimate.
4. Adjust `toolBaseCosts` values based on observed usage and re-run.

Cycles reservations lock the estimated amount and commits charge the same estimate, so the values only need to be "close enough" — precise per-call cost reconciliation is reserved for consumers who import `cycles-openclaw-budget-guard` as an npm library in custom agent frameworks.
:::

## Tool access control

Control which tools can be called using allowlists and blocklists with glob-style patterns:

```json
{
  "plugins": {
    "entries": {
      "openclaw-budget-guard": {
        "config": {
          "tenant": "acme",
          "toolAllowlist": ["web_search", "code_*"],
          "toolBlocklist": ["dangerous_*"]
        }
      }
    }
  }
}
```

- Blocklist takes precedence over allowlist
- Supports exact names and `*` wildcards anywhere in the pattern (prefix: `code_*`, suffix: `*_tool`, mid: `aws_*_tool`, all: `*`)
- Tools blocked by access lists are rejected before any budget reservation is attempted

## Tool call limits

Cap the number of times a specific tool can be invoked per session. Useful for consequential actions like sending emails or triggering deployments:

```json
{
  "plugins": {
    "entries": {
      "openclaw-budget-guard": {
        "config": {
          "tenant": "acme",
          "toolCallLimits": {
            "send_email": 10,
            "deploy": 3
          }
        }
      }
    }
  }
}
```

Once a tool reaches its limit, further calls are blocked with a descriptive reason. Tools without a limit are unrestricted. Limits reset on each new agent session.

::: tip Combine with cost limits
Tool call limits complement cost-based budgeting. Use `toolBaseCosts` to control spend and `toolCallLimits` to cap side-effects independently — an agent can exhaust its budget before hitting call limits, or vice versa.
:::

## Graceful degradation strategies

When budget is low, the plugin can apply multiple composable strategies beyond model downgrading:

```json
{
  "plugins": {
    "entries": {
      "openclaw-budget-guard": {
        "config": {
          "tenant": "acme",
          "lowBudgetStrategies": ["downgrade_model", "reduce_max_tokens", "disable_expensive_tools"],
          "maxTokensWhenLow": 1024,
          "expensiveToolThreshold": 1000000
        }
      }
    }
  }
}
```

Available strategies:

| Strategy | Effect |
|----------|--------|
| `downgrade_model` | Use cheaper fallback models from `modelFallbacks` (default) |
| `reduce_max_tokens` | Append token limit guidance to prompt hints |
| `disable_expensive_tools` | Block tools exceeding `expensiveToolThreshold` |
| `limit_remaining_calls` | Cap total tool/model calls via `maxRemainingCallsWhenLow` (default: 10) |

Strategies are composable — list multiple values to combine them.

## Prompt budget hints

When `injectPromptBudgetHint` is enabled (the default), the plugin prepends a compact hint to the system prompt so the model itself is aware of budget constraints:

```
Budget: 5000000 USD_MICROCENTS remaining. Budget is low — prefer cheaper models and avoid expensive tools. 50% of budget remaining. Est. ~10 tool calls and ~5 model calls remaining at current rate. Team pool: 50000000 remaining.
```

The hint includes:
- Current remaining balance and percentage
- Budget level warnings
- Forecast projections based on average call costs so far
- Team pool balance (when `parentBudgetId` is configured)
- Token limit guidance (when `reduce_max_tokens` strategy is active)

This helps models self-regulate — choosing cheaper tools, shorter responses, or skipping optional steps when budget is tight.

## Per-user and per-session attribution

Attach user and session identifiers to reservations for attribution and reporting:

```json
{
  "plugins": {
    "entries": {
      "openclaw-budget-guard": {
        "config": {
          "tenant": "acme",
          "userId": "user-123",
          "sessionId": "session-456"
        }
      }
    }
  }
}
```

User and session identifiers can also be set dynamically via `ctx.metadata.userId` and `ctx.metadata.sessionId` at runtime — context values override static config.

These identifiers are threaded into Cycles reservation subjects as `dimensions` for attribution and reporting. Note that dimensions never derive budget scopes — to *enforce* a per-user or per-session budget, the identifier must be mapped to a standard Subject field (e.g. `agent` or `workflow`) with a budget at that scope.

## Reservation settings

Configure reservation behavior per-tool or globally:

```json
{
  "plugins": {
    "entries": {
      "openclaw-budget-guard": {
        "config": {
          "tenant": "acme",
          "reservationTtlMs": 60000,
          "toolReservationTtls": {
            "code_execution": 120000
          },
          "overagePolicy": "ALLOW_IF_AVAILABLE",
          "toolOveragePolicies": {
            "web_search": "ALLOW_IF_AVAILABLE"
          }
        }
      }
    }
  }
}
```

Overage policies control what happens when a reservation exceeds the remaining budget:
- `ALLOW_IF_AVAILABLE` — allow up to the remaining balance (default)
- `REJECT` — deny the reservation
- `ALLOW_WITH_OVERDRAFT` — allow and create a debt

## Retry on denied reservations

Optionally retry tool reservations that are denied, useful when budget is being replenished or released concurrently:

```json
{
  "plugins": {
    "entries": {
      "openclaw-budget-guard": {
        "config": {
          "tenant": "acme",
          "retryOnDeny": true,
          "retryDelayMs": 2000,
          "maxRetries": 1
        }
      }
    }
  }
}
```

::: warning Latency note
Retries block the `before_tool_call` hook synchronously. With the defaults (`retryDelayMs: 2000`, `maxRetries: 1`), a denied tool call pauses for 2 seconds before returning a block result. For interactive agents where responsiveness matters, consider lowering `retryDelayMs` or keeping `retryOnDeny: false` (the default) and handling denials at the application level instead.
:::

## Budget transition alerts

Get notified when the budget level changes (e.g., healthy → low → exhausted):

```json
{
  "plugins": {
    "entries": {
      "openclaw-budget-guard": {
        "config": {
          "tenant": "acme",
          "budgetTransitionWebhookUrl": "https://hooks.example.com/budget-alert"
        }
      }
    }
  }
}
```

The webhook POST body carries `{ previousLevel, currentLevel, remaining, timestamp }`. Receiver services should be idempotent and ack quickly — delivery is fire-and-forget (best-effort, no retries).

::: info Note
Transition detection runs on every budget snapshot refresh (controlled by `snapshotCacheTtlMs`, default 5 seconds). If a budget oscillates rapidly around a threshold between cache refreshes, the same transition (e.g., healthy → low) may fire more than once. Webhook receivers should be idempotent or deduplicate by timestamp if this matters for your use case.
:::

## Session analytics and cost breakdown

The plugin tracks per-tool and per-model cost breakdowns throughout the session. At `agent_end`, it builds a `SessionSummary` containing:

- Tenant, budget, user, and session identifiers
- Final remaining/spent/reserved balances
- Total reservations made
- Per-component cost breakdown (e.g., `tool:web_search`, `model:anthropic/claude-sonnet-4-6`)
- Per-tool invocation counts (e.g., `{ web_search: 15, code_execution: 3 }`)
- Session timing (start/end timestamps)
- Average cost and estimated remaining calls

The summary is attached to `ctx.metadata["openclaw-budget-guard"]` and can also be exported via webhook:

```json
{
  "plugins": {
    "entries": {
      "openclaw-budget-guard": {
        "config": {
          "tenant": "acme",
          "analyticsWebhookUrl": "https://analytics.example.com/sessions"
        }
      }
    }
  }
}
```

The webhook receives the full `SessionSummary` as its POST body, so the receiving service can persist or forward it to any analytics backend.

## End-user budget visibility

Budget status is automatically attached to `ctx.metadata["openclaw-budget-guard-status"]` on every hook invocation, making it available to OpenClaw frontends for UI display:

```json
{
  "level": "low",
  "remaining": 5000000,
  "allocated": 10000000,
  "percentRemaining": 50
}
```

## Multi-currency support

Override the default currency per-tool or per-model:

```json
{
  "plugins": {
    "entries": {
      "openclaw-budget-guard": {
        "config": {
          "tenant": "acme",
          "currency": "USD_MICROCENTS",
          "modelCurrency": "TOKENS",
          "toolCurrencies": {
            "web_search": "CREDITS"
          }
        }
      }
    }
  }
}
```

Each reservation uses the appropriate currency unit. Cost tracking respects the per-component currency.

## Budget pools

Surface hierarchical budget information by setting a parent budget ID:

```json
{
  "plugins": {
    "entries": {
      "openclaw-budget-guard": {
        "config": {
          "tenant": "acme",
          "budgetScope": { "app": "team-alpha-agent" },
          "parentBudgetId": "team-alpha"
        }
      }
    }
  }
}
```

When `parentBudgetId` is set, the matching balance is included in budget snapshots and prompt hints (for example, "Team pool: 50000000 remaining."). This setting is read-only visibility: reservations still target `budgetScope`, and `parentBudgetId` does not add another enforced ledger. To enforce a broader limit as well, create a budget at a broader standard scope that is actually present in the reservation subject.

## Dry-run mode

Test the plugin without a live Cycles server:

```json
{
  "plugins": {
    "entries": {
      "openclaw-budget-guard": {
        "config": {
          "tenant": "acme",
          "cyclesBaseUrl": "http://unused",
          "cyclesApiKey": "unused",
          "dryRun": true,
          "dryRunBudget": 100000000
        }
      }
    }
  }
}
```

In dry-run mode, budget is tracked in-memory using a simulated client. All plugin behavior (classification, reservation, fallbacks, strategies) works identically — only the Cycles server communication is replaced. This is useful for development, testing, and evaluating the plugin before deploying a Cycles server.

## Fail-open vs fail-closed

The plugin distinguishes between two failure modes:

**Budget confirmed exhausted** — controlled by `failClosed` (default: `true`):
- `failClosed: true` → returns OpenClaw's blocking result for tools; for model calls, overrides the model with the plugin's non-existent exhaustion sentinel so the provider rejects before generation
- `failClosed: false` → logs a warning but allows execution to continue

**Budget snapshot unavailable** — controlled by `failClosedOnSnapshotError` (default: `false`):
- `failClosedOnSnapshotError: false` → assumes a healthy snapshot and continues to the reservation step
- `failClosedOnSnapshotError: true` → treats the snapshot as exhausted; `failClosed` then decides whether to block

**Reservation unavailable or denied** — the reservation helper returns a denial after its configured transient retries. `failClosed: true` blocks the call; `failClosed: false` logs the denial and allows execution while tracking the estimate locally.

## Error handling

The package exports two structured error classes for custom integrations:

```typescript
import { BudgetExhaustedError, ToolBudgetDeniedError } from "@runcycles/openclaw-budget-guard";
```

- **`BudgetExhaustedError`** (`code: "BUDGET_EXHAUSTED"`) — includes `remaining`, `tenant`, and `budgetId` properties.
- **`ToolBudgetDeniedError`** (`code: "TOOL_BUDGET_DENIED"`) — includes `toolName`.

The built-in hooks do not currently throw these classes. `before_tool_call` returns `{ block: true, blockReason }` for a fail-closed denial, while `before_model_resolve` returns `modelOverride: "__cycles_budget_exhausted__"` because OpenClaw does not expose a model-call block result.

## Verifying the integration

Set `logLevel: "debug"` to see the plugin's activity:

```json
{
  "plugins": {
    "entries": {
      "openclaw-budget-guard": {
        "config": {
          "tenant": "acme",
          "logLevel": "debug"
        }
      }
    }
  }
}
```

On startup, the plugin logs a config summary so you can verify settings at a glance:

```
  Cycles Budget Guard for OpenClaw v0.8.4
  https://runcycles.io
  tenant: acme
  cyclesBaseUrl: http://localhost:7878
  cyclesApiKey: ****_key
  currency: USD_MICROCENTS
  failClosed: true
  dryRun: false
  logLevel: debug
  lowBudgetThreshold: 10000000
  exhaustedThreshold: 0
```

The plugin also warns about common misconfigurations on startup (e.g., `downgrade_model` strategy with no `modelFallbacks`, or no `toolBaseCosts` configured).

With `logLevel: "debug"`, you'll see per-call activity:

```
[openclaw-budget-guard] before_model_resolve: model=anthropic/claude-sonnet-4-6 level=healthy
[openclaw-budget-guard] before_prompt_build: injecting hint (142 chars)
[openclaw-budget-guard] Tool "web_search" has no entry in toolBaseCosts — using default estimate (100000 USD_MICROCENTS)
[openclaw-budget-guard] before_tool_call: tool=web_search callId=abc123 estimate=100000
[openclaw-budget-guard] after_tool_call: committed 100000 for tool=web_search
[openclaw-budget-guard] Agent session budget summary: remaining=9500000 spent=500000 reservations=1
```

## Full configuration reference

### Core settings

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `enabled` | boolean | `true` | Master switch |
| `cyclesBaseUrl` | string | — | Cycles server URL (required) |
| `cyclesApiKey` | string | — | Cycles API key (required) |
| `tenant` | string | — | Cycles tenant (required) |
| `budgetScope` | object | — | Scope segments for targeting a specific budget (e.g. `{ "workspace": "road", "app": "lane" }`) |
| `budgetId` | string | — | **Deprecated** — use `budgetScope`. Equivalent to `budgetScope: { "app": "<value>" }` |
| `currency` | string | `USD_MICROCENTS` | Default budget unit |
| `failClosed` | boolean | `true` | Block on exhausted snapshots and reservation denials |
| `failClosedOnSnapshotError` | boolean | `false` | Treat a failed or timed-out budget snapshot fetch as exhausted instead of healthy |
| `logLevel` | string | `info` | `debug` / `info` / `warn` / `error` |

### Budget thresholds

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `lowBudgetThreshold` | number | `10000000` | Below this → low budget mode |
| `exhaustedThreshold` | number | `0` | At or below this → exhausted |

### Model configuration

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `modelFallbacks` | object | `{}` | Model → fallback model or chain (string or string[]) |
| `modelBaseCosts` | object | `{}` | Model name → estimated cost per call |
| `defaultModelCost` | number | `500000` | Fallback cost when model not in `modelBaseCosts` |
| `defaultModelName` | string | — | Model name for budget reservations. Required because OpenClaw's `before_model_resolve` event doesn't include the model name. Set to your agent's model (e.g. `"openai/gpt-5-nano"`). |
| `defaultModelActionKind` | string | `llm.completion` | Action kind for model reservations |
| `modelCurrency` | string | — | Override currency for model reservations |

### Tool configuration

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `toolBaseCosts` | object | `{}` | Tool name → estimated cost per call |
| `defaultToolActionKindPrefix` | string | `tool.` | Prefix for tool action kinds |
| `toolAllowlist` | string[] | — | Only these tools are permitted (supports `*` wildcards) |
| `toolBlocklist` | string[] | — | These tools are blocked (supports `*` wildcards) |
| `toolCurrencies` | object | — | Tool name → currency override |
| `toolReservationTtls` | object | — | Tool name → TTL override (ms) |
| `toolOveragePolicies` | object | — | Tool name → overage policy override |
| `toolCallLimits` | object | — | Map: tool name → max invocations per session |

### Prompt hints

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `injectPromptBudgetHint` | boolean | `true` | Inject budget status into the system prompt |
| `maxPromptHintChars` | number | `200` | Max characters for the budget hint |

### Reservation settings

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `reservationTtlMs` | number | `60000` | Default reservation TTL (ms) |
| `overagePolicy` | string | `ALLOW_IF_AVAILABLE` | Default overage policy (`REJECT`, `ALLOW_IF_AVAILABLE`, `ALLOW_WITH_OVERDRAFT`) |
| `snapshotCacheTtlMs` | number | `5000` | Budget snapshot cache TTL (ms) |

### Low-budget strategies

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `lowBudgetStrategies` | string[] | `["downgrade_model"]` | Strategies to apply when budget is low |
| `maxTokensWhenLow` | number | `1024` | Token limit when `reduce_max_tokens` is active |
| `expensiveToolThreshold` | number | — | Cost threshold for `disable_expensive_tools` |
| `maxRemainingCallsWhenLow` | number | `10` | Max calls when `limit_remaining_calls` is active |

### Retry on deny

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `retryOnDeny` | boolean | `false` | Retry tool reservations after denial |
| `retryDelayMs` | number | `2000` | Delay between retries (ms) |
| `maxRetries` | number | `1` | Maximum retry attempts |

### Dry-run mode

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `dryRun` | boolean | `false` | Use in-memory simulated budget |
| `dryRunBudget` | number | `100000000` | Starting budget for dry-run mode |

### Per-user/session attribution

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `userId` | string | — | User ID recorded in `dimensions` for attribution/reporting — not budget-enforcing (overridable via `ctx.metadata.userId`) |
| `sessionId` | string | — | Session ID recorded in `dimensions` for attribution/reporting — not budget-enforcing (overridable via `ctx.metadata.sessionId`) |

### Budget transitions

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `budgetTransitionWebhookUrl` | string | — | Webhook URL for level transitions |

### Session analytics

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `analyticsWebhookUrl` | string | — | Webhook URL for session summary data |

### Budget pools

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `parentBudgetId` | string | — | Parent budget ID for pool balance visibility |

### Advanced / operational settings

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `aggressiveCacheInvalidation` | boolean | `true` | Invalidate budget snapshot cache after every mutation |
| `heartbeatIntervalMs` | number | `30000` | Interval for reservation TTL heartbeat extensions |
| `retryableStatusCodes` | number[] | `[429, 503, 504]` | HTTP status codes eligible for transient retry |
| `transientRetryMaxAttempts` | number | `2` | Max retries on transient server errors |
| `transientRetryBaseDelayMs` | number | `500` | Base delay for transient retries (ms) |
| `burnRateWindowMs` | number | `60000` | Window for burn-rate anomaly detection |
| `burnRateAlertThreshold` | number | `3.0` | Alert when burn rate exceeds this multiple of average |
| `enableEventLog` | boolean | `false` | Log all budget events for debugging |
| `exhaustionWarningThresholdMs` | number | `120000` | Warn when time-to-exhaustion falls below this (ms) |
| `otlpMetricsEndpoint` | string | — | OpenTelemetry OTLP endpoint for budget metrics |
| `otlpMetricsHeaders` | object | — | Custom headers for OTLP exporter |

## Comparison with manual integration

If you're already using the Cycles TypeScript client directly (see [Programmatic Client Usage](/how-to/using-the-cycles-client-programmatically)), the plugin automates the same reserve → commit → release pattern but at the OpenClaw hook level:

| Concern | Manual client | OpenClaw plugin |
|---------|--------------|-----------------|
| Reserve before LLM call | Your code | `before_model_resolve` hook |
| Reserve before tool call | Your code | `before_tool_call` hook |
| Commit after completion | Your code | `after_tool_call` hook |
| Release orphans | Your code | `agent_end` hook |
| Model downgrade on low budget | Your code | Automatic via `modelFallbacks` |
| Prompt budget awareness | Your code | Automatic via `injectPromptBudgetHint` |
| Cost breakdown tracking | Your code | Automatic per-tool/model tracking |
| Session analytics | Your code | Automatic via `analyticsWebhookUrl` |
| Tool access control | Your code | Automatic via `toolAllowlist` / `toolBlocklist` |

The plugin is the recommended approach for OpenClaw users — it requires zero custom code and covers the full lifecycle automatically.

## Config presets

Common starting configurations for typical deployment scenarios.

### Strict enforcement

For production agents handling real spend. Blocks on exhaustion, downgrades models, caps tool calls:

```json
{
  "plugins": {
    "entries": {
      "openclaw-budget-guard": {
        "config": {
          "tenant": "my-org",
          "failClosed": true,
          "lowBudgetStrategies": ["downgrade_model", "disable_expensive_tools", "limit_remaining_calls"],
          "modelFallbacks": {
            "anthropic/claude-opus-4-8": ["anthropic/claude-sonnet-4-6", "anthropic/claude-haiku-4-5-20251001"]
          },
          "modelBaseCosts": {
            "anthropic/claude-opus-4-8": 500000,
            "anthropic/claude-sonnet-4-6": 300000,
            "anthropic/claude-haiku-4-5-20251001": 100000
          },
          "toolBaseCosts": {
            "web_search": 500000,
            "code_execution": 1000000
          },
          "toolCallLimits": {
            "send_email": 10,
            "deploy": 3
          },
          "maxRemainingCallsWhenLow": 5
        }
      }
    }
  }
}
```

### Development / testing

Dry-run mode with generous budget. No Cycles server needed:

```json
{
  "plugins": {
    "entries": {
      "openclaw-budget-guard": {
        "config": {
          "tenant": "dev",
          "cyclesBaseUrl": "http://unused",
          "cyclesApiKey": "unused",
          "dryRun": true,
          "dryRunBudget": 500000000,
          "logLevel": "debug"
        }
      }
    }
  }
}
```

### Cost-conscious

Aggressive cost savings. Low thresholds, model downgrade with token limits, expensive tools disabled early:

```json
{
  "plugins": {
    "entries": {
      "openclaw-budget-guard": {
        "config": {
          "tenant": "my-org",
          "lowBudgetThreshold": 5000000,
          "exhaustedThreshold": 100000,
          "lowBudgetStrategies": ["downgrade_model", "reduce_max_tokens", "disable_expensive_tools"],
          "maxTokensWhenLow": 512,
          "expensiveToolThreshold": 200000,
          "modelFallbacks": {
            "anthropic/claude-opus-4-8": "anthropic/claude-haiku-4-5-20251001",
            "openai/gpt-4o": "openai/gpt-4o-mini"
          }
        }
      }
    }
  }
}
```

## Model cost reconciliation (v0.5.0)

Model costs are estimated at reservation time based on `modelBaseCosts` or `defaultModelCost`. Since OpenClaw doesn't provide token counts after model completion, exact per-call costs aren't available through the JSON-configured OpenClaw path.

v0.5.0 introduces the **reserve-then-commit** pattern for models: the plugin reserves budget in `before_model_resolve` but doesn't commit until the next `before_prompt_build` (or `agent_end` for the last turn). The reservation is committed at the estimated cost.

```json
{
  "config": {
    "modelBaseCosts": {
      "anthropic/claude-sonnet-4-6": 300000,
      "anthropic/claude-opus-4-8": 500000
    }
  }
}
```

Tune the numbers iteratively: enable `enableEventLog: true`, run representative sessions, and adjust `modelBaseCosts` using the per-model cost breakdown in the session summary. Reservations lock the estimate and commits charge the same value, so close-enough estimates are sufficient for enforcement.

::: info Precise per-call reconciliation
Token-accurate reconciliation via a `modelCostEstimator` callback is available only when consuming `cycles-openclaw-budget-guard` as an npm library in a custom agent framework — OpenClaw's JSON plugin config cannot carry JavaScript functions.
:::

## Observability with OTLP metrics (v0.5.0)

The plugin can emit structured metrics to any observability backend (Datadog, Prometheus, Grafana, OpenTelemetry) via the built-in OTLP HTTP adapter.

### Using the built-in OTLP adapter

For zero-config OpenTelemetry integration, set `otlpMetricsEndpoint`:

```json
{
  "config": {
    "otlpMetricsEndpoint": "http://localhost:4318/v1/metrics",
    "otlpMetricsHeaders": {
      "Authorization": "Bearer <token>"
    }
  }
}
```

The plugin auto-creates a lightweight OTLP HTTP adapter that buffers metrics and flushes them periodically. No OpenTelemetry SDK dependency required.

::: info Custom emitter
A programmatic `metricsEmitter` interface (`gauge` / `counter` / `histogram`) is available only when consuming `cycles-openclaw-budget-guard` as an npm library — OpenClaw's JSON plugin config cannot carry JavaScript functions. For OpenClaw deployments, point `otlpMetricsEndpoint` at your collector (most observability backends, including Datadog and Prometheus via a receiver, accept OTLP).
:::

### Emitted metrics

| Metric | Type | Tags | When |
|--------|------|------|------|
| `cycles.budget.remaining` | gauge | tenant, budgetScope keys, currency | Every snapshot fetch |
| `cycles.budget.reserved` | gauge | tenant, budgetScope keys | Every snapshot fetch |
| `cycles.budget.spent` | gauge | tenant, budgetScope keys | Every snapshot fetch |
| `cycles.budget.level` | gauge (0/1/2) | tenant, budgetScope keys, level | Every snapshot fetch |
| `cycles.reservation.created` | counter | tenant, kind, name | On reserve |
| `cycles.reservation.committed` | counter | tenant, kind, name | On commit |
| `cycles.reservation.denied` | counter | tenant, kind, name, reason | On deny |
| `cycles.reservation.cost` | histogram | tenant, kind, name | On commit |
| `cycles.model.downgrade` | counter | tenant, from, to | On model downgrade |
| `cycles.tool.blocked` | counter | tenant, tool, reason | On tool block |
| `cycles.session.duration_ms` | histogram | tenant | On agent_end |
| `cycles.session.total_cost` | histogram | tenant | On agent_end |
| `cycles.budget.burn_rate_anomaly` | counter | tenant, ratio | On burn rate spike (v0.6.0) |
| `cycles.budget.exhaustion_forecast_ms` | gauge | tenant | On exhaustion forecast (v0.6.0) |

## Aggressive cache invalidation (v0.5.0)

By default (`aggressiveCacheInvalidation: true`), the plugin refetches the budget snapshot from the Cycles server after every commit or release. This reduces the "stale window" from the `snapshotCacheTtlMs` (default 5s) to near-zero for single-agent scenarios.

For high-throughput setups where the extra network call is undesirable, disable it:

```json
{
  "config": {
    "aggressiveCacheInvalidation": false,
    "snapshotCacheTtlMs": 2000
  }
}
```

## Resilience: retry and heartbeat (v0.6.0)

### Automatic retry on transient errors

The plugin retries Cycles server requests on transient HTTP errors (429, 503, 504) with exponential backoff:

```json
{
  "config": {
    "retryableStatusCodes": [429, 503, 504],
    "transientRetryMaxAttempts": 2,
    "transientRetryBaseDelayMs": 500
  }
}
```

With default settings, a 429 response triggers up to 2 retries with 500ms and 1000ms delays. The plugin builds one request body and idempotency key for the logical reservation, then reuses both across those transport retries. A successful retry therefore resolves to the same idempotent reservation operation rather than creating a second hold.

### Heartbeat for long-running tools

Tools that run longer than the reservation TTL (default 60s) previously lost cost tracking silently. v0.6.0 auto-extends reservations:

```json
{
  "config": {
    "heartbeatIntervalMs": 30000
  }
}
```

Every 30 seconds, the plugin calls the Cycles `extend` endpoint to keep the reservation alive. The timer is automatically stopped when the tool completes or at `agent_end`. Set to `0` to disable.

## Anomaly detection (v0.6.0)

### Burn rate monitoring

Detect runaway tool loops by monitoring cost-per-window:

```json
{
  "config": {
    "burnRateWindowMs": 60000,
    "burnRateAlertThreshold": 3.0
  }
}
```

If the cost rate in the current window exceeds 3x the previous window, the plugin emits the `cycles.budget.burn_rate_anomaly` counter (tagged with `currentBurnRate`, `averageBurnRate`, `ratio`, `threshold`, and `remaining`). Wire alerts to this metric in your OTLP-connected backend (Datadog, Prometheus, Grafana) to page on runaway loops.

### Predictive exhaustion warning

Get advance notice before budget runs out:

```json
{
  "config": {
    "exhaustionWarningThresholdMs": 120000
  }
}
```

When estimated time-to-exhaustion drops below 120 seconds (based on current burn rate), the plugin emits the `cycles.budget.exhaustion_forecast_ms` gauge. The warning fires once per session. Alert on this gauge in your OTLP-connected backend to trigger a top-up or graceful wind-down.

## Session event log (v0.6.0)

Enable a session log of the plugin's budget decisions:

```json
{
  "config": {
    "enableEventLog": true
  }
}
```

When enabled, `sessionSummary.eventLog` contains every reserve, commit, deny, block, and release event with timestamps, budget levels, and amounts. The log is capped at 10,000 entries. Useful for debugging budget exhaustion and understanding agent behavior.

Example event:
```json
{
  "timestamp": 1711468850000,
  "hook": "before_tool_call",
  "action": "reserve",
  "kind": "tool",
  "name": "web_search",
  "amount": 500000,
  "decision": "ALLOW",
  "budgetLevel": "healthy",
  "remaining": 45000000
}
```

The session summary also includes `unconfiguredTools` — a list of tools that used the default cost estimate (100,000 units) because they had no entry in `toolBaseCosts`. Use this to identify configuration gaps.

## Model blocking workaround (v0.7.3)

OpenClaw's `before_model_resolve` hook does not support `{ block: true }` like `before_tool_call` does ([feature request](https://github.com/openclaw/openclaw/issues/55771)). When budget is exhausted, the plugin cannot directly prevent the model call.

**Workaround:** The plugin returns `{ modelOverride: "__cycles_budget_exhausted__" }`, which causes the LLM provider to reject the request with "Unknown model." The agent receives no response and no budget is spent.

The user sees:
```
Agent failed before reply: Unknown model: openai/__cycles_budget_exhausted__
```

This is intentional. When OpenClaw adds `block` support to `before_model_resolve`, the plugin will switch to a clean blocking mechanism with a proper error message.

**Note:** OpenClaw's `before_model_resolve` event also does not include the model name — it only passes `{ prompt }`. Set `defaultModelName` in your plugin config so the plugin knows which model to track:

```json
{
  "config": {
    "defaultModelName": "openai/gpt-5-nano"
  }
}
```

## What to do when budget is exhausted

1. **Fund the budget** via the Cycles Admin API:
   ```bash
   curl -X POST "http://localhost:7979/v1/admin/budgets/fund?scope=tenant:my-org&unit=USD_MICROCENTS" \
     -H "X-Cycles-API-Key: your-tenant-key" \
     -H "Content-Type: application/json" \
     -d '{"operation": "CREDIT", "amount": {"amount": 50000000, "unit": "USD_MICROCENTS"}, "idempotency_key": "topup-001"}'
   ```
   This adds 50,000,000 units ($0.50) to the budget. Adjust the `scope` to match your `tenant` and `budgetScope`.

2. **Start a new agent session** — the plugin fetches fresh budget state at the start of each session.

For details, see [Budget Allocation and Management](/how-to/budget-allocation-and-management-in-cycles).

## Troubleshooting

**"Skipping registration" warning during install**
- This is normal. OpenClaw loads the plugin during install before your config is written. The plugin detects the missing config, logs a warning, and skips registration. After you add your config and restart the gateway, the plugin will register normally.

**Plugin not loading**
- Verify the plugin is enabled: `openclaw plugins list`
- Check that `openclaw.plugin.json` is included in the installed package

**"cyclesBaseUrl is required" error**
- Set `cyclesBaseUrl` in your plugin config (use `"${CYCLES_BASE_URL}"` for env var interpolation)

**"tenant is required" error**
- Add `"tenant": "your-org"` to the plugin config

**Budget always shows "healthy"**
- Verify `currency`, `tenant`, and `budgetScope` match your Cycles setup
- Set `logLevel: "debug"` to see raw balance responses

**Tools not being blocked**
- Check `toolBaseCosts` includes your tool (default cost is 100,000 units)
- Check `failClosed` is `true` (default)

**"No toolBaseCosts configured" warning**
- This is informational. Without `toolBaseCosts`, all tools use the default cost estimate (100,000 units). Add entries for your tools to improve budgeting accuracy.

**Model not being downgraded**
- The exact model name must match a key in `modelFallbacks`
- Check model costs in `modelBaseCosts` — fallback must be cheaper than remaining budget
- If you see "no modelFallbacks configured" warning, add a `modelFallbacks` entry

## Next steps

- [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) — strategies beyond simple model downgrade
- [Estimate Exposure Before Execution](/how-to/how-to-estimate-exposure-before-execution-practical-reservation-strategies-for-cycles) — how to set `toolBaseCosts` effectively
- [Shadow Mode](/how-to/shadow-mode-in-cycles-how-to-roll-out-budget-enforcement-without-breaking-production) — roll out enforcement without breaking production
- [Runaway Agents and Tool Loops](/incidents/runaway-agents-tool-loops-and-budget-overruns-the-incidents-cycles-is-designed-to-prevent) — the incidents this plugin helps prevent



# Integrating Cycles with Pydantic AI

This guide shows how to guard [Pydantic AI](https://pydantic.dev/docs/ai/overview/) agent runs with Cycles budget reservations so that every agent invocation is cost-controlled and observable.

## Prerequisites

```bash
pip install runcycles pydantic-ai
```

Set environment variables:

```bash
export CYCLES_BASE_URL="http://localhost:7878"
export CYCLES_API_KEY="your-api-key"   # create via Admin Server — see note below
export CYCLES_TENANT="acme"
export OPENAI_API_KEY="sk-..."         # or whichever provider your agent uses
```

> **Need an API key?** Create one via the Admin Server — see [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack#step-3-create-an-api-key) or [API Key Management](/how-to/api-key-management-in-cycles).

> **Note:** The snippets on this page target pydantic-ai 1.x (`result.output`, `output_type=`, `input_tokens`/`output_tokens`). Pre-1.0 releases used `result.data`, `result_type=`, and `request_tokens`/`response_tokens`.

::: tip 60-Second Quick Start
```python
from pydantic_ai import Agent
from runcycles import CyclesClient, CyclesConfig, cycles, set_default_client

set_default_client(CyclesClient(CyclesConfig.from_env()))

agent = Agent("openai:gpt-4o", system_prompt="You are a helpful assistant.")

@cycles(estimate=1_500_000, action_kind="llm.completion", action_name="gpt-4o")
def ask(prompt: str) -> str:
    result = agent.run_sync(prompt)
    return result.output

print(ask("What is budget authority?"))
```
Every call is now budget-guarded. If the budget is exhausted, `BudgetExceededError` is raised _before_ the agent runs. Read on for production patterns with tool calls and structured output.
:::

## Guarding model calls

Use `@cycles` to wrap an agent run with automatic reserve, execute, and commit:

```python
from pydantic_ai import Agent
from runcycles import (
    CyclesClient, CyclesConfig, CyclesMetrics,
    cycles, get_cycles_context, set_default_client,
)

set_default_client(CyclesClient(CyclesConfig.from_env()))

PRICE_PER_INPUT_TOKEN = 250       # $2.50 / 1M tokens in microcents
PRICE_PER_OUTPUT_TOKEN = 1_000    # $10.00 / 1M tokens in microcents

agent = Agent("openai:gpt-4o", system_prompt="You are a research assistant.")

@cycles(
    estimate=2_000_000,
    actual=lambda result: (
        result["usage"]["input_tokens"] * PRICE_PER_INPUT_TOKEN
        + result["usage"]["output_tokens"] * PRICE_PER_OUTPUT_TOKEN
    ),
    action_kind="llm.completion",
    action_name="gpt-4o",
    unit="USD_MICROCENTS",
    ttl_ms=60_000,
)
def research(question: str) -> dict:
    result = agent.run_sync(question)

    ctx = get_cycles_context()
    if ctx:
        ctx.metrics = CyclesMetrics(
            tokens_input=result.usage().input_tokens,
            tokens_output=result.usage().output_tokens,
        )

    return {
        "answer": result.output,
        "usage": {
            "input_tokens": result.usage().input_tokens,
            "output_tokens": result.usage().output_tokens,
        },
    }
```

## Tool call budget scoping

When your Pydantic AI agent uses tools, you can budget each tool invocation separately by wrapping tool functions with `@cycles`:

```python
from pydantic_ai import Agent, RunContext

agent = Agent("openai:gpt-4o", system_prompt="Use tools to answer questions.")

@agent.tool
@cycles(estimate=50_000, action_kind="tool.search", action_name="web-search")
def search_web(ctx: RunContext[None], query: str) -> str:
    """Search the web for information."""
    # Your search implementation
    return perform_search(query)

@agent.tool
@cycles(estimate=20_000, action_kind="tool.lookup", action_name="db-lookup")
def lookup_database(ctx: RunContext[None], record_id: str) -> str:
    """Look up a record in the database."""
    return db.get(record_id)

@cycles(estimate=2_000_000, action_kind="llm.completion", action_name="gpt-4o")
def ask_with_tools(prompt: str) -> str:
    result = agent.run_sync(prompt)
    return result.output
```

Each tool call gets its own reservation, so you have fine-grained visibility into what the agent spends on LLM calls versus tool invocations.

## Structured output with budget control

Pydantic AI excels at returning structured data. Combine this with Cycles to budget-guard typed responses:

```python
from pydantic import BaseModel
from pydantic_ai import Agent
from runcycles import cycles

class MovieReview(BaseModel):
    title: str
    rating: float
    summary: str

review_agent = Agent(
    "openai:gpt-4o",
    output_type=MovieReview,
    system_prompt="You are a film critic. Return structured reviews.",
)

@cycles(estimate=1_500_000, action_kind="llm.completion", action_name="gpt-4o")
def review_movie(movie_name: str) -> MovieReview:
    result = review_agent.run_sync(f"Review the movie: {movie_name}")
    return result.output
```

The decorator does not interfere with the return type. Your function still returns a `MovieReview` instance; Cycles only manages the budget lifecycle around it.

## Error handling

When the budget is insufficient, `BudgetExceededError` is raised **before** the agent runs:

```python
from runcycles import BudgetExceededError

try:
    answer = ask("Summarize recent ML papers")
except BudgetExceededError:
    answer = "Budget exhausted — please try again later."
```

For agents with tool calls, each tool decorated with `@cycles` can independently raise `BudgetExceededError`. Handle this at the outer call to catch failures at any level:

```python
try:
    result = ask_with_tools("Find the latest sales data and summarize it")
except BudgetExceededError as e:
    print(f"Budget limit hit: {e}")
    result = fallback_response()
```

See [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) for patterns like queueing, model downgrade, and caching.

## Key points

- **Decorator wraps any function.** The `@cycles` decorator works with `agent.run_sync()`, async runs, and tool functions alike.
- **Tool-level budgets.** Decorate individual `@agent.tool` functions for per-tool cost visibility.
- **Structured output is preserved.** Cycles does not alter your function's return type or Pydantic models.
- **The agent never runs on DENY.** If the budget is exhausted, the LLM is never called, saving both cost and latency.

## Next steps

- [Error Handling Patterns in Python](/how-to/error-handling-patterns-in-python) — handling budget errors in Python
- [Handling Streaming Responses](/how-to/handling-streaming-responses-with-cycles) — budget-managed streaming
- [Testing with Cycles](/how-to/testing-with-cycles) — testing budget-guarded code
- [Production Operations Guide](/how-to/production-operations-guide) — running Cycles in production
- [Integrating with OpenAI](/how-to/integrating-cycles-with-openai) — if your Pydantic AI agent uses OpenAI models
- [Integrating with Anthropic](/how-to/integrating-cycles-with-anthropic) — if your Pydantic AI agent uses Anthropic models



# Rust AI Agent Budget Control — Cycles Integration Guide

If you're building AI agents or LLM-powered services in Rust, hard spending limits and tool-call governance are critical *before* execution, not after. This guide shows how to guard Rust async operations with Cycles budget reservations — from one-liner wrappers to full manual control — covering the patterns Tokio-native applications need: streaming responses, RAII safety, Axum / Actix middleware, and multi-step agent workflows.

Same protocol as the Python, TypeScript, and Spring Boot clients, so the same Cycles server works across polyglot stacks.

## Prerequisites

```toml
# Cargo.toml
[dependencies]
runcycles = "0.3"
tokio = { version = "1", features = ["full"] }
```

Set environment variables:

```bash
export CYCLES_BASE_URL="http://localhost:7878"
export CYCLES_API_KEY="your-api-key"
export CYCLES_TENANT="acme"
```

> **Need an API key?** Create one via the Admin Server — see [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack#step-3-create-an-api-key) or [API Key Management](/how-to/api-key-management-in-cycles).

> **Looking for a real LLM example?** The `openai_call` / `stream_llm_response` placeholders below are intentionally generic. For the concrete `async-openai` wiring — including streaming token capture and error mapping — see [Integrate Cycles with async-openai (Rust)](/how-to/integrating-cycles-with-async-openai).

## Quick start

```rust
use runcycles::{CyclesClient, CyclesConfig, with_cycles, WithCyclesConfig, models::Amount};

#[tokio::main]
async fn main() -> Result<(), runcycles::Error> {
    let client = CyclesClient::new(CyclesConfig::from_env()?);

    let reply = with_cycles(
        &client,
        WithCyclesConfig::new(Amount::tokens(1000))
            .action("llm.completion", "gpt-4o"),
        |ctx| async move {
            let result = call_llm("What is budget authority?").await?;
            Ok((result, Amount::tokens(42)))
        },
    ).await?;

    println!("{reply}");
    Ok(())
}
```

`with_cycles` handles the full lifecycle: reserve → execute → commit on success, release on error.

## Three integration levels

### Level 1: `with_cycles()` — automatic lifecycle

The simplest option. Equivalent to Python's `@cycles` decorator:

```rust
use runcycles::{with_cycles, WithCyclesConfig, models::*};

let config = WithCyclesConfig::new(Amount::usd_microcents(2_000_000))
    .action("llm.completion", "gpt-4o")
    .subject(Subject { tenant: Some("acme".into()), ..Default::default() });

let result = with_cycles(&client, config, |ctx| async move {
    // ctx.decision — Allow or AllowWithCaps
    // ctx.caps — soft constraints (max_tokens, tool_denylist, etc.)
    // ctx.reservation_id — for logging

    if let Some(caps) = &ctx.caps {
        if let Some(max_tokens) = caps.max_tokens {
            // Respect server-imposed token limits
        }
    }

    let response = openai_call(&prompt).await?;
    let actual_cost = Amount::usd_microcents(1_800_000);
    Ok((response, actual_cost))
}).await?;
```

The closure receives a `GuardContext` and must return `Result<(T, Amount), Box<dyn Error>>` — the value plus the actual cost for commit.

### Level 2: `ReservationGuard` — RAII manual control

For streaming, multi-step workflows, or when you need to inspect the guard between reserve and commit:

```rust
use runcycles::models::*;

let guard = client.reserve(
    ReservationCreateRequest::builder()
        .subject(Subject { tenant: Some("acme".into()), ..Default::default() })
        .action(Action::new("llm.completion", "gpt-4o"))
        .estimate(Amount::usd_microcents(5_000_000))
        .build()
).await?;

// Check decision
match guard.decision() {
    Decision::Allow => { /* full access */ }
    Decision::AllowWithCaps => {
        // Adapt to caps
        if let Some(caps) = guard.caps() {
            println!("max_tokens: {:?}", caps.max_tokens);
        }
    }
}

// Execute the operation
let (response, actual_tokens) = stream_llm_response(&prompt).await?;

// Commit actual cost (consumes guard — compile-time move safety)
guard.commit(
    CommitRequest::builder()
        .actual(Amount::usd_microcents(actual_tokens))
        .build()
).await?;
```

If the guard is dropped without commit or release (panic, early `?` return), it auto-releases via `Drop` — no leaked reservations.

### Level 3: `CyclesClient` — programmatic API

Full control over every protocol operation:

```rust
use runcycles::models::*;

// Reserve
let res = client.create_reservation(&ReservationCreateRequest::builder()
    .subject(Subject { tenant: Some("acme".into()), ..Default::default() })
    .action(Action::new("tool.call", "search"))
    .estimate(Amount::tokens(500))
    .build()
).await?;

let reservation_id = res.reservation_id
    .expect("ALLOW decision always includes reservation_id");

// Execute
let result = do_work().await;

// Commit or release
match result {
    Ok(value) => {
        client.commit_reservation(
            &reservation_id,
            &CommitRequest::builder()
                .actual(Amount::tokens(320))
                .build()
        ).await?;
    }
    Err(_) => {
        client.release_reservation(
            &reservation_id,
            &ReleaseRequest::new(Some("operation_failed".into()))
        ).await?;
    }
}
```

## RISK_POINTS for action control

Guard non-monetary actions using risk-point budgets:

```rust
use runcycles::models::*;

// Reserve risk points instead of dollars
let guard = client.reserve(
    ReservationCreateRequest::builder()
        .subject(Subject { tenant: Some("acme".into()), ..Default::default() })
        .action(Action::new("tool.email", "send_customer_email"))
        .estimate(Amount::risk_points(50))
        .build()
).await?;

// Execute the action
send_email(&recipient, &body).await?;

// Commit
guard.commit(
    CommitRequest::builder()
        .actual(Amount::risk_points(50))
        .build()
).await?;
```

See [Action Authority](/concepts/action-authority-controlling-what-agents-do) for the full risk-point model.

## Axum middleware

Wrap all routes with budget enforcement:

```rust
use axum::{extract::State, middleware, Router};
use runcycles::{CyclesClient, with_cycles, WithCyclesConfig, models::Amount};

async fn budget_layer(
    State(client): State<CyclesClient>,
    req: axum::http::Request<axum::body::Body>,
    next: middleware::Next,
) -> Result<axum::response::Response, AppError> {
    let config = WithCyclesConfig::new(Amount::usd_microcents(500_000))
        .action("http.request", req.uri().path());

    let response = with_cycles(&client, config, |_ctx| async move {
        let resp = next.run(req).await;
        Ok((resp, Amount::usd_microcents(300_000)))
    }).await?;

    Ok(response)
}

let app = Router::new()
    .route("/chat", axum::routing::post(chat_handler))
    .layer(middleware::from_fn_with_state(client.clone(), budget_layer))
    .with_state(client);
```

## Multi-tenant routing

Extract tenant from request headers and scope budgets per-tenant:

```rust
use axum::extract::State;
use axum::http::HeaderMap;
use runcycles::{CyclesClient, with_cycles, WithCyclesConfig, models::*};

async fn chat(
    State(client): State<CyclesClient>,
    headers: HeaderMap,
    body: String,
) -> Result<String, AppError> {
    let tenant = headers.get("X-Tenant-ID")
        .and_then(|v| v.to_str().ok())
        .ok_or(AppError::missing_tenant())?;

    let config = WithCyclesConfig::new(Amount::usd_microcents(2_000_000))
        .action("llm.completion", "gpt-4o")
        .subject(Subject { tenant: Some(tenant.into()), ..Default::default() });

    let reply = with_cycles(&client, config, |_ctx| async move {
        let result = call_llm(&body).await?;
        Ok((result, Amount::usd_microcents(1_500_000)))
    }).await?;

    Ok(reply)
}
```

## Environment-based configuration

```rust
use runcycles::{CyclesClient, CyclesConfig};

// From environment variables (CYCLES_BASE_URL, CYCLES_API_KEY, etc.)
let client = CyclesClient::new(CyclesConfig::from_env()?);

// From builder
let client = CyclesClient::builder("cyc_live_abc123", "http://localhost:7878")
    .tenant("acme")
    .workspace("prod")
    .connect_timeout(std::time::Duration::from_secs(2))
    .read_timeout(std::time::Duration::from_secs(5))
    .retry_enabled(true)
    .retry_max_attempts(5)
    .build();
```

See the [Rust Client Configuration Reference](/configuration/rust-client-configuration-reference) for the full surface.

## Blocking client

For synchronous Rust applications (not using tokio). The blocking client uses `create_reservation` / `commit_reservation` / `release_reservation` directly — no `ReservationGuard` (guards require async for heartbeat and Drop):

```rust
use runcycles::{CyclesClient, models::*};

let client = CyclesClient::builder("cyc_live_abc123", "http://localhost:7878")
    .tenant("acme")
    .build_blocking()?;

let res = client.create_reservation(&ReservationCreateRequest::builder()
    .subject(Subject { tenant: Some("acme".into()), ..Default::default() })
    .action(Action::new("llm.completion", "gpt-4o"))
    .estimate(Amount::usd_microcents(2_000_000))
    .build()
)?;

let reservation_id = res.reservation_id
    .expect("ALLOW decision always includes reservation_id");

// ... do work ...

client.commit_reservation(
    &reservation_id,
    &CommitRequest::builder()
        .actual(Amount::usd_microcents(1_500_000))
        .build()
)?;
```

Enable with:

```toml
[dependencies]
runcycles = { version = "0.3", features = ["blocking"] }
```

## Error handling

See [Error Handling in Rust](/how-to/error-handling-patterns-in-rust) for comprehensive patterns including:

- `Error::BudgetExceeded` — DENY handling with retry delay
- `ReservationGuard` RAII safety — compile-time double-commit prevention
- Axum `IntoResponse` error handler
- Transient vs non-transient error table

## Next steps

- [Getting Started with the Rust Client](/quickstart/getting-started-with-the-rust-client) — full quickstart with all three integration levels
- [Error Handling in Rust](/how-to/error-handling-patterns-in-rust) — comprehensive error patterns
- [How to Add Budget and Action Guardrails to Rust AI Agents](/blog/how-to-add-budget-and-action-guardrails-to-rust-ai-agents-with-cycles) — end-to-end agent example
- [Action Authority](/concepts/action-authority-controlling-what-agents-do) — RISK_POINTS for controlling what agents do
- [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) — strategies for handling budget constraints



# Integrating Cycles with Spring AI

This guide shows how to guard Spring AI chat completions and tool calls with Cycles budget reservations so that every LLM interaction is cost-controlled, caps-aware, and observable.

For strategic guidance on where to integrate, see [Budget Limits with Spring AI](/quickstart/how-to-add-hard-budget-limits-to-spring-ai-with-cycles).

## Two integration paths

Cycles ships two complementary Java starters. Pick based on your call surface:

| Aspect | [`cycles-spring-ai-starter`](https://github.com/runcycles/cycles-spring-ai-starter) | [`cycles-spring-boot-starter`](https://github.com/runcycles/cycles-spring-boot-starter) |
|---|---|---|
| Maven artifact | `io.runcycles:cycles-spring-ai-starter` | `io.runcycles:cycles-client-java-spring` |
| Mechanism | Spring AI `CallAdvisor` + `StreamAdvisor` + `ChatClientCustomizer` (auto-wired); `CyclesToolGate` for per-tool gating | Spring AOP via `@Cycles` annotation |
| Where it intercepts | Every `chatClient.prompt(...).call()` and `.stream()` invocation; per-tool when wrapped via `cyclesToolGate.wrap(...)` | Any Java method you annotate |
| Call-site changes | **No** — transparent wiring for chat (tool wrapping is opt-in) | Yes — add `@Cycles` annotation |
| Estimate computation | Pluggable `PromptTokenEstimator`: chars/4 heuristic by default, real BPE via jtokkit (opt-in) or custom bean | SpEL expression: `@Cycles("#tokens * 250")` |
| Subject routing | Pluggable `SubjectResolver`: property defaults, or per-call (e.g. tenant from `SecurityContextHolder`) via custom bean | SpEL: can pull tenant from method args |
| Knows about LLMs? | Yes — Spring AI ChatClient specific | No — generic for any cost-incurring code |

**Use [`cycles-spring-ai-starter`](#path-1-auto-wired-advisor-cycles-spring-ai-starter)** if your LLM calls go through Spring AI's `ChatClient`.

**Use [`cycles-spring-boot-starter`](#path-2-cycles-annotation-cycles-client-java-spring)** for non-Spring-AI code paths (custom HTTP clients, LangChain4j, vector store queries, etc.) — or when you need SpEL-driven per-method estimates.

::: warning Don't double-charge
Wrapping a Spring AI chat call inside an `@Cycles`-annotated method produces **two reservations** for one operation — once from the AOP wrapper, once from the Spring AI advisor. Pick one strategy per call path. See the "Double-charge gotcha" section in the [`cycles-spring-ai-starter` README](https://github.com/runcycles/cycles-spring-ai-starter).
:::

---

## Path 1: Auto-wired advisor (`cycles-spring-ai-starter`)

The simplest path for Spring AI apps — add the dependency, configure a few `cycles.*` properties, and every `ChatClient.call()` and `.stream()` invocation is auto-gated.

### 1. Add the dependency

::: code-group
```xml [Maven]
<dependency>
    <groupId>io.runcycles</groupId>
    <artifactId>cycles-spring-ai-starter</artifactId>
    <version>0.4.0</version>
</dependency>
```
```groovy [Gradle]
implementation 'io.runcycles:cycles-spring-ai-starter:0.4.0'
```
:::

This transitively pulls in `cycles-client-java-spring` for the HTTP client to the Cycles server.

### 2. Configure

```yaml
cycles:
  base-url: http://localhost:7878
  api-key:  ${CYCLES_API_KEY}
  tenant:   acme
  app:      my-spring-ai-app
  spring-ai:
    enabled: true
    default-estimate: 1000          # micro-cents per call; set estimate-from-prompt=true to derive from prompt size
    estimate-unit: USD_MICROCENTS
    action-kind: llm.chat
    action-name: spring-ai-chat
    fail-open: false                # true = log + proceed on Cycles errors
```

### 3. Use ChatClient normally

```java
@Service
public class OrderAgent {
    private final ChatClient chatClient;

    public OrderAgent(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }

    public String summarize(String order) {
        // Cycles reserves budget BEFORE this call hits the LLM.
        // If the budget is exhausted, CyclesBudgetDeniedException is thrown
        // and the LLM call never happens.
        return chatClient.prompt()
                .user("Summarize: " + order)
                .call()
                .content();
    }
}
```

No annotations. No `@Cycles`. The advisor is auto-attached to every `ChatClient` built from the auto-configured `ChatClient.Builder` via a `ChatClientCustomizer`.

### What v0.3.0 covers

Everything v0.2.0 shipped is still here — drop-in compatible — plus three new extension points and a trace-correlation tag.

✅ **Non-streaming `.call()`** — full reserve → call → commit (on success) / release (on exception) lifecycle. Deny throws `CyclesBudgetDeniedException` before the LLM is contacted.

✅ **Streaming `.stream()`** — `CyclesBudgetStreamAdvisor` mirrors the lifecycle for `chatClient.prompt(...).stream()` invocations. Per-subscription reservation (wrapped in `Flux.defer`); commits on successful completion using usage from the last chunk; releases on stream error or subscriber cancellation. Reserve and commit failures surface as `onError` to the subscriber, matching reactive-idiomatic shape and the fail-fast contract of the non-streaming advisor.

✅ **Real `ChatResponse.Usage` extraction on commit** — when the LLM provider returns usage:
- `cycles.spring-ai.estimate-unit=TOKENS`: commits `Usage.getTotalTokens()` directly.
- `input-cost-per-token` and/or `output-cost-per-token` set: commits `(promptTokens × inputRate) + (completionTokens × outputRate)`.
- Otherwise (no rates, no TOKENS unit): commits the estimate as actual (v0.1.0-compatible fallback).

When both token breakdowns are null (provider returned a placeholder `Usage` with no breakdown), falls back to the estimate rather than under-billing with a zero commit.

✅ **Prompt-based per-call estimate** — `cycles.spring-ai.estimate-from-prompt=true` with at least one cost-per-token rate set derives the pre-call reservation amount from the configured `PromptTokenEstimator`. Default is a `chars / 4` heuristic; set `cycles.spring-ai.token-estimator-encoding=cl100k_base` (or `o200k_base`) + add the jtokkit dep to opt into real BPE encoding (see below). Falls back to `default-estimate` when the prompt is empty or rates are zero. Applies to both the call and stream advisors.

✅ **Tool-level gating via `CyclesToolGate`** — auto-configured factory bean. Wrap any Spring AI `ToolCallback` with `cyclesToolGate.wrap(myTool)` to gate per-tool invocations through Cycles. Tool reservations report distinct action labels (`tool.call` / `spring-ai-tool:<tool-name>` by default — configurable) so they're separable from chat reservations in audit history. Opt-in: Spring AI doesn't provide a hook to auto-decorate every registered tool.

✅ **`CyclesChatClientObservationConvention`** — extends Spring AI's `DefaultChatClientObservationConvention` and appends low-cardinality Cycles attribution tags to every chat-client trace: `cycles.tenant`, `cycles.workspace`, `cycles.app`, `cycles.action_kind`, `cycles.action_name`. **New in 0.3.0:** also emits `cycles.reservation_id` as a high-cardinality `KeyValue` for trace ↔ reservation correlation in your tracing backend. Auto-configured as a bean but not auto-attached — apply explicitly via `builder.observationConvention(cyclesConvention)`. Disable the high-cardinality tag with `cycles.spring-ai.emit-reservation-id-on-trace=false` if your tracing backend charges by unique tag-value combinations.

#### New extension points in v0.3.0

**Pluggable `SubjectResolver`** — multi-tenant agents need per-request attribution. By default the starter reads tenant/workspace/app from `CyclesProperties` on every call (every reservation is attributed to the same subject). Register a `SubjectResolver` bean for per-call routing:

```java
@Bean
public SubjectResolver tenantAwareSubjectResolver(CyclesProperties defaults) {
    return request -> {
        var auth = SecurityContextHolder.getContext().getAuthentication();
        String tenant = (auth != null && auth.isAuthenticated()) ? auth.getName() : defaults.getTenant();
        return Subject.builder()
                .tenant(tenant)
                .workspace(defaults.getWorkspace())
                .app(defaults.getApp())
                .build();
    };
}
```

`@ConditionalOnMissingBean` ensures your bean wins over the property-derived default. The `request` parameter is `null` on the tool-gating path (tool callbacks don't carry a `ChatClientRequest`); implementations should handle `null` defensively.

**Pluggable `PromptTokenEstimator` with jtokkit** — v0.2.0 hard-coded prompt-token estimation as `chars / 4`. v0.3.0 makes it pluggable and ships a real BPE impl via [jtokkit](https://github.com/knuddelsgmbh/jtokkit). Opt in:

```yaml
cycles:
  spring-ai:
    estimate-from-prompt: true
    input-cost-per-token: 250                  # 1 USD = 100,000,000 USD_MICROCENTS, so $2.50/1M tokens = 250 microcents/token
    output-cost-per-token: 1000                # $10.00/1M tokens = 1000 microcents/token
    token-estimator-encoding: o200k_base   # gpt-4o family; cl100k_base for gpt-4 / gpt-3.5-turbo
```

```xml
<dependency>
    <groupId>com.knuddels</groupId>
    <artifactId>jtokkit</artifactId>
    <version>1.1.0</version>
</dependency>
```

The jtokkit dep is `optional=true` on the starter — only opt-in users pay the size cost. Setting the property without the dep on the classpath logs a WARN at app startup and falls back to chars/4. For provider-specific tokenizers, register your own `PromptTokenEstimator` bean.

See the [Spring AI Starter Configuration Reference](/configuration/spring-ai-starter-configuration-reference) for every property, auto-configuration condition, extension point, and failure-mode boundary.

---

## Path 2: `@Cycles` annotation (`cycles-client-java-spring`)

Use this path when:
- Your LLM calls go through code that is **not** Spring AI's `ChatClient` (custom HTTP, LangChain4j, in-house wrappers).
- You need **dynamic per-call estimates** via SpEL expressions.
- You want **explicit control** over which methods are gated.

## Prerequisites

Add the Cycles Spring Boot Starter to your project:

::: code-group
```xml [Maven]
<dependency>
    <groupId>io.runcycles</groupId>
    <artifactId>cycles-client-java-spring</artifactId>
    <version>0.3.2</version>
</dependency>
```
```groovy [Gradle]
implementation 'io.runcycles:cycles-client-java-spring:0.3.2'
```
:::

Configure the connection in `application.yml`:

```yaml
cycles:
  base-url: http://localhost:7878
  api-key: ${CYCLES_API_KEY}
  tenant: acme
  app: my-spring-ai-app
```

> **Need an API key?** Create one via the Admin Server — see [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack#step-3-create-an-api-key) or [API Key Management](/how-to/api-key-management-in-cycles).

::: tip 60-Second Quick Start
```java
import io.runcycles.client.java.spring.annotation.Cycles;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.stereotype.Service;

@Service
public class ChatService {

    private final ChatClient chatClient;

    public ChatService(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }

    // GPT-4o: ~$2.50/1M input tokens = 250 microcents/token
    @Cycles(value = "#maxTokens * 250",
            actionKind = "llm.completion",
            actionName = "gpt-4o")
    public String chat(String prompt, int maxTokens) {
        return chatClient.prompt(prompt)
            .call()
            .content();
    }
}
```

That's it. Every call to `chat()` is now budget-guarded: Cycles reserves the estimated cost before execution, commits actual usage after, and throws `CyclesProtocolException` if the budget is exceeded.
:::

## Dynamic cost estimation with Spring AI

Use SpEL expressions to estimate cost from method parameters. The `value` (or `estimate`) attribute is evaluated before the method runs:

```java
// Estimate based on max tokens × price per token (in USD_MICROCENTS)
// GPT-4o: ~$2.50/1M input tokens = 250 microcents/token
@Cycles(value = "#maxTokens * 250",
        actionKind = "llm.completion",
        actionName = "gpt-4o")
public String generate(String prompt, int maxTokens) {
    return chatClient.prompt(prompt)
        .call()
        .content();
}

// Estimate from prompt length (rough token approximation: ~4 chars per token)
@Cycles(value = "#prompt.length() / 4 * 250",
        actionKind = "llm.completion",
        actionName = "gpt-4o")
public String summarize(String prompt) {
    return chatClient.prompt(prompt)
        .call()
        .content();
}
```

See [SpEL Expression Reference](/configuration/spel-expression-reference-for-cycles) for all available expressions.

## Reporting actual usage

The `actual` attribute is evaluated after the method returns, using `#result` to reference the return value. This lets Cycles commit the real cost instead of the estimate:

```java
@Cycles(value = "#maxTokens * 250",
        actual = "#result.length() / 4 * 250",
        actionKind = "llm.completion",
        actionName = "gpt-4o")
public String generate(String prompt, int maxTokens) {
    return chatClient.prompt(prompt)
        .call()
        .content();
}
```

For precise token counts, access the `ChatResponse` metadata and report via `CyclesMetrics`:

```java
import io.runcycles.client.java.spring.annotation.Cycles;
import io.runcycles.client.java.spring.context.CyclesContextHolder;
import io.runcycles.client.java.spring.context.CyclesReservationContext;
import io.runcycles.client.java.spring.model.CyclesMetrics;

@Cycles(value = "#maxTokens * 250",
        actionKind = "llm.completion",
        actionName = "gpt-4o")
public String generateWithMetrics(String prompt, int maxTokens) {
    long start = System.currentTimeMillis();

    ChatResponse response = chatClient.prompt(prompt)
        .call()
        .chatResponse();

    String content = response.getResult().getOutput().getText();

    // Report exact token usage via the reservation context
    CyclesReservationContext ctx = CyclesContextHolder.get();
    if (ctx != null) {
        Usage usage = response.getMetadata().getUsage();
        CyclesMetrics metrics = new CyclesMetrics();
        metrics.setTokensInput((int) usage.getPromptTokens());
        metrics.setTokensOutput((int) usage.getCompletionTokens());
        metrics.setLatencyMs((int) (System.currentTimeMillis() - start));
        metrics.setModelVersion("gpt-4o-2024-08-06");
        ctx.setMetrics(metrics);
    }

    return content;
}
```

The `actual` SpEL attribute on `@Cycles` handles cost calculation. Use `CyclesMetrics` for observability data (token counts, latency, model version) that is attached to the commit for reporting.

## Respecting budget caps in Spring AI

When the deepest matching budget has caps configured, Cycles can return `ALLOW_WITH_CAPS` instead of a flat `ALLOW`. This is configuration-driven, not an automatic low-balance transition. Read the caps from the reservation context and apply them — for example, by reducing max tokens:

```java
@Cycles(value = "#maxTokens * 250",
        actionKind = "llm.completion",
        actionName = "gpt-4o")
public String capsAwareChat(String prompt, int maxTokens) {
    CyclesReservationContext ctx = CyclesContextHolder.get();

    // Respect token cap from budget authority
    int effectiveMaxTokens = maxTokens;
    if (ctx != null && ctx.hasCaps() && ctx.getCaps().getMaxTokens() != null) {
        effectiveMaxTokens = Math.min(maxTokens, ctx.getCaps().getMaxTokens());
    }

    return chatClient.prompt(prompt)
        .options(ChatOptions.builder()
            .maxTokens(effectiveMaxTokens)
            .build())
        .call()
        .content();
}
```

## Error handling

Catch `CyclesProtocolException` to degrade gracefully when budget is exceeded. This should be part of your service layer from the start:

```java
import io.runcycles.client.java.spring.model.CyclesProtocolException;

@Service
public class ResilientChatService {

    private final GuardedLlmService premiumLlm;
    private final GuardedLlmService budgetLlm;

    public String chat(String prompt) {
        try {
            return premiumLlm.generate(prompt, 4096);     // GPT-4o
        } catch (CyclesProtocolException e) {
            if (e.isBudgetExceeded()) {
                return budgetLlm.generate(prompt, 1024);   // GPT-4o-mini fallback
            }
            if (e.getRetryAfterMs() != null) {
                scheduleRetry(prompt, e.getRetryAfterMs());
                return "Request queued. Retrying shortly.";
            }
            throw e;
        }
    }
}
```

`GuardedLlmService` is a separate `@Service` bean whose methods are annotated with `@Cycles`. This is needed because Spring AOP proxies only intercept calls from outside the bean — see [Self-invocation workaround](#self-invocation-workaround) below.

For global exception handling in a REST API:

```java
@ControllerAdvice
public class CyclesExceptionHandler {

    @ExceptionHandler(CyclesProtocolException.class)
    public ResponseEntity<Map<String, Object>> handleBudgetError(CyclesProtocolException e) {
        if (e.isBudgetExceeded()) {
            return ResponseEntity.status(429)
                .header("Retry-After", String.valueOf(
                    e.getRetryAfterMs() != null ? e.getRetryAfterMs() / 1000 : 60))
                .body(Map.of("error", "budget_exceeded", "message", "Budget limit reached."));
        }
        return ResponseEntity.status(503)
            .body(Map.of("error", e.getReasonCode(), "message", e.getMessage()));
    }
}
```

## Guarding Spring AI tool calls

For Spring AI function callbacks, wrap the tool execution with `@Cycles` on a separate service bean:

```java
@Service
public class GuardedToolService {

    @Cycles(value = "500000",  // $0.005 per tool call
            actionKind = "tool.search",
            actionName = "web-search",
            toolset = "search-tools")
    public String webSearch(String query) {
        return searchApi.search(query);
    }

    @Cycles(value = "100000",  // $0.001 per DB query
            actionKind = "tool.database",
            actionName = "sql-query",
            toolset = "data-tools")
    public String queryDatabase(String sql) {
        return jdbcTemplate.queryForList(sql).toString();
    }
}
```

Then register these as Spring AI tool callbacks:

```java
import org.springframework.ai.tool.ToolCallback;
import org.springframework.ai.tool.function.FunctionToolCallback;

@Configuration
public class ToolConfig {

    @Bean
    public ToolCallback webSearchTool(GuardedToolService tools) {
        return FunctionToolCallback.builder("web_search", (String query) -> tools.webSearch(query))
            .description("Search the web")
            .inputType(String.class)
            .build();
    }
}
```

The `toolset` attribute scopes budget per tool category, so you can set different budgets for search tools vs. database tools via the Admin API.

## Spring AI streaming with budget control

For streaming, use the programmatic `CyclesClient` instead of the annotation, since the stream needs to commit after all chunks arrive:

```java
import io.runcycles.client.java.spring.client.CyclesClient;
import io.runcycles.client.java.spring.model.*;

@Service
public class StreamingChatService {

    private final ChatClient chatClient;
    private final CyclesClient cyclesClient;

    public Flux<String> streamChat(String prompt, int maxTokens) {
        // Reserve budget before streaming
        Map<String, Object> body = Map.of(
            "idempotency_key", UUID.randomUUID().toString(),
            "subject", Map.of("tenant", "acme"),
            "action", Map.of("kind", "llm.completion", "name", "gpt-4o"),
            "estimate", Map.of("unit", "USD_MICROCENTS", "amount", maxTokens * 250L),
            "ttl_ms", 120000
        );

        var response = cyclesClient.createReservation(body);
        String reservationId = response.getBodyAttributeAsString("reservation_id");
        String decision = response.getBodyAttributeAsString("decision");

        if (!"ALLOW".equals(decision) && !"ALLOW_WITH_CAPS".equals(decision)) {
            throw new CyclesProtocolException("Budget denied: " + decision);
        }

        AtomicInteger tokenCount = new AtomicInteger();

        return chatClient.prompt(prompt)
            .stream()
            .content()
            .doOnNext(chunk -> tokenCount.addAndGet(chunk.length() / 4))
            .doOnComplete(() -> {
                cyclesClient.commitReservation(reservationId, Map.of(
                    "idempotency_key", UUID.randomUUID().toString(),
                    "actual", Map.of("unit", "USD_MICROCENTS",
                                     "amount", tokenCount.get() * 250L)
                ));
            })
            .doOnError(err -> {
                cyclesClient.releaseReservation(reservationId, Map.of(
                    "idempotency_key", UUID.randomUUID().toString(),
                    "reason", "stream_error: " + err.getMessage()
                ));
            });
    }
}
```

## Agent loop budget control

For multi-step agent workflows, guard each iteration. Each call gets its own reservation, so Cycles can deny mid-workflow when budget runs out:

```java
@Service
public class AgentService {

    private final GuardedLlmService llm;

    public String runAgent(String task, int maxIterations) {
        String context = task;

        for (int i = 0; i < maxIterations; i++) {
            try {
                String response = llm.generate(context, 2048);
                if (isComplete(response)) {
                    return response;
                }
                context = response;
            } catch (CyclesProtocolException e) {
                if (e.isBudgetExceeded()) {
                    return "Agent stopped: budget exhausted after " + i + " iterations.";
                }
                throw e;
            }
        }

        return "Agent reached max iterations.";
    }
}
```

## Production patterns

### Dry-run rollout

Start in shadow mode to measure budget impact before enforcing:

```java
@Cycles(value = "#maxTokens * 250",
        actionKind = "llm.completion",
        actionName = "gpt-4o",
        dryRun = true)
public Object shadowChat(String prompt, int maxTokens) {  // returns DryRunResult, not the chat content
    return chatClient.prompt(prompt).call().content();
}
```

::: warning
When `dryRun = true`, the guarded method does **not** execute. The annotation evaluates the reservation against the budget but skips method execution and returns a framework result object. Use this to measure what budget impact would be, not for serving production traffic.
:::

### Multi-tenant via SpEL

Resolve tenant from the method parameters:

```java
@Cycles(value = "#maxTokens * 250",
        tenant = "#tenantId",
        actionKind = "llm.completion",
        actionName = "gpt-4o")
public String tenantChat(String tenantId, String prompt, int maxTokens) {
    return chatClient.prompt(prompt).call().content();
}
```

### Self-invocation workaround

Spring AOP proxies do not intercept self-calls within the same bean. If you call an `@Cycles` method from another method in the same class, the annotation is bypassed. Use a separate service bean:

```java
// This bean's @Cycles annotations ARE intercepted by the proxy
@Service
public class GuardedLlmService {
    private final ChatClient chatClient;

    public GuardedLlmService(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }

    @Cycles(value = "#maxTokens * 250", actionKind = "llm.completion", actionName = "gpt-4o")
    public String generate(String prompt, int maxTokens) {
        return chatClient.prompt(prompt).call().content();
    }
}

// This bean calls the guarded bean — proxy intercepts correctly
@Service
public class AgentOrchestrator {
    @Autowired private GuardedLlmService llm;

    public String orchestrate(String task) {
        return llm.generate(task, 2048);  // @Cycles is applied
    }
}
```

## Key points

- `@Cycles` works with any Spring AI `ChatClient` or `ChatModel` call — no adapter needed
- Use `value` (SpEL) to estimate cost before execution, `actual` to commit real cost after
- `CyclesContextHolder.get()` provides reservation context inside the guarded method — use it for caps and metrics
- Guard tool calls with `@Cycles` on a separate `@Service` bean, scoped with `toolset`
- For streaming, use the programmatic `CyclesClient` instead of the annotation
- Catch `CyclesProtocolException` to degrade to a cheaper model or queue for retry
- Start with `dryRun = true` for shadow-mode rollouts before enforcing

## Next steps

For **Path 1 (`cycles-spring-ai-starter`):**

- [Spring AI Starter Configuration Reference](/configuration/spring-ai-starter-configuration-reference) — every property, extension point, auto-configuration condition, and the double-charge boundary
- [`cycles-spring-ai-starter` README](https://github.com/runcycles/cycles-spring-ai-starter) — source-repository quickstart and examples
- [`cycles-spring-ai-starter` on Maven Central](https://central.sonatype.com/artifact/io.runcycles/cycles-spring-ai-starter)
- [Budget Limits with Spring AI](/quickstart/how-to-add-hard-budget-limits-to-spring-ai-with-cycles) — strategic guidance on where to put the gates

For **Path 2 (`@Cycles` annotation / `cycles-client-java-spring`):**

- [Spring Boot Starter Quickstart](/quickstart/getting-started-with-the-cycles-spring-boot-starter) — demo app, annotation reference, full walkthrough
- [Spring Client Configuration](/configuration/client-configuration-reference-for-cycles-spring-boot-starter) — all `cycles.*` properties
- [SpEL Expression Reference](/configuration/spel-expression-reference-for-cycles) — estimate and actual expressions
- [Choosing the Right Overage Policy](/how-to/choosing-the-right-overage-policy) — REJECT vs ALLOW_IF_AVAILABLE vs ALLOW_WITH_OVERDRAFT



# Integrating Cycles with the Vercel AI SDK

This guide shows how to add budget governance to a Next.js application using the [Vercel AI SDK](https://ai-sdk.dev/) and the `runcycles` TypeScript client.

The Vercel AI SDK uses streaming by default, so this guide uses the `reserveForStream` pattern — reserving budget before the stream starts, keeping the reservation alive during streaming, and committing actual usage when the stream finishes.

## Prerequisites

- A running Cycles stack with a tenant, API key, and budget ([Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack))
- A Next.js project with the Vercel AI SDK installed
- Node.js 20+

## Installation

```bash
npm install runcycles ai@^4 @ai-sdk/openai@^1
```

This guide targets AI SDK v4 (the major pinned by the SDK's [example project](https://github.com/runcycles/cycles-client-typescript/tree/main/examples/vercel-ai-sdk); note the example's route code is currently being aligned to the same v4 surface). AI SDK 5 renames several of these APIs (`usage.inputTokens`/`outputTokens`, `toUIMessageStreamResponse()`, `maxOutputTokens`, `useChat` from `@ai-sdk/react`) — adjust accordingly if you are on v5.

## Environment variables

```bash
CYCLES_BASE_URL=http://localhost:7878
CYCLES_API_KEY=cyc_live_...
CYCLES_TENANT=acme-corp
OPENAI_API_KEY=sk-...
```

::: tip 60-Second Quick Start
```typescript
import { streamText } from "ai";
import { openai } from "@ai-sdk/openai";
import { CyclesClient, CyclesConfig, reserveForStream } from "runcycles";

const cycles = new CyclesClient(CyclesConfig.fromEnv());
const handle = await reserveForStream({
  client: cycles, estimate: 2_000_000, unit: "USD_MICROCENTS",
  actionKind: "llm.completion", actionName: "gpt-4o",
});

const result = streamText({
  model: openai("gpt-4o"),
  prompt: "What is budget authority?",
  onFinish: async ({ usage }) => {
    await handle.commit((usage.promptTokens ?? 0) * 250 + (usage.completionTokens ?? 0) * 1000);
  },
});
```
Budget is reserved before the stream starts and committed when it finishes. Read on for the full Next.js API route pattern with error handling.
:::

## API route with budget governance

Create an API route that reserves budget before streaming and commits actual usage after:

```typescript
// app/api/chat/route.ts
import { streamText, type Message, convertToCoreMessages } from "ai";
import { openai } from "@ai-sdk/openai";
import {
  CyclesClient,
  CyclesConfig,
  reserveForStream,
  BudgetExceededError,
} from "runcycles";

export const runtime = "nodejs"; // Required for AsyncLocalStorage

const cyclesClient = new CyclesClient(CyclesConfig.fromEnv());

export async function POST(req: Request) {
  const { messages }: { messages: Message[] } = await req.json();

  // Estimate cost from message content (1 token ~ 4 chars).
  // GPT-4o: input $2.50/1M tokens (250 microcents/token),
  //         output $10/1M tokens (1000 microcents/token).
  const estimatedInputTokens = messages.reduce(
    (sum, m) => sum + (typeof m.content === "string" ? m.content.length : 0) / 4,
    0,
  );
  const estimatedCost = Math.ceil(
    estimatedInputTokens * 250 + estimatedInputTokens * 2 * 1000,
  );

  // 1. Reserve budget
  let handle;
  try {
    handle = await reserveForStream({
      client: cyclesClient,
      estimate: estimatedCost,
      unit: "USD_MICROCENTS",
      actionKind: "llm.completion",
      actionName: "gpt-4o",
    });
  } catch (err) {
    if (err instanceof BudgetExceededError) {
      return new Response(
        JSON.stringify({
          error: "budget_exceeded",
          message: "Budget exhausted. Contact your administrator.",
        }),
        { status: 402, headers: { "Content-Type": "application/json" } },
      );
    }
    throw err;
  }

  // 2. Stream with budget tracking
  try {
    const result = streamText({
      model: openai("gpt-4o"),
      messages: convertToCoreMessages(messages),
      onFinish: async ({ usage }) => {
        const actualCost = Math.ceil(
          (usage.promptTokens ?? 0) * 250 +
          (usage.completionTokens ?? 0) * 1000,
        );
        await handle.commit(actualCost, {
          tokensInput: usage.promptTokens,
          tokensOutput: usage.completionTokens,
        });
      },
    });

    return result.toDataStreamResponse();
  } catch (err) {
    await handle.release("stream_error");
    throw err;
  }
}
```

## How it works

1. **Before streaming:** `reserveForStream` creates a reservation and starts an automatic heartbeat to keep it alive during the stream.
2. **During streaming:** The Vercel AI SDK streams tokens to the client. The heartbeat extends the reservation TTL automatically.
3. **After streaming:** The `onFinish` callback calculates actual cost from token usage and calls `handle.commit()`. The heartbeat stops automatically.
4. **On error:** The `catch` block calls `handle.release()` to return the reserved budget to the pool.

## Respecting budget caps

When the deepest matching budget has `max_tokens` configured, Cycles can return `ALLOW_WITH_CAPS`. This is not an automatic low-balance transition. Respect the returned value by capping the model's output:

```typescript
let handle = await reserveForStream({ ... });

// Use caps-aware max_tokens
let maxTokens = 4096;
if (handle.caps?.maxTokens) {
  maxTokens = Math.min(maxTokens, handle.caps.maxTokens);
}

const result = streamText({
  model: openai("gpt-4o"),
  maxTokens,
  messages: convertToCoreMessages(messages),
  onFinish: async ({ usage }) => { ... },
});
```

## Client-side error handling

Handle the 402 response in your React component:

```typescript
// components/chat.tsx
import { useChat } from "ai/react";

export function Chat() {
  const { messages, input, handleInputChange, handleSubmit, error } = useChat();

  if (error?.message?.includes("budget_exceeded")) {
    return <div>Your budget has been exhausted. Please contact support.</div>;
  }

  return (
    <form onSubmit={handleSubmit}>
      {messages.map((m) => (
        <div key={m.id}>{m.content}</div>
      ))}
      <input value={input} onChange={handleInputChange} />
    </form>
  );
}
```

## Next steps

- [Integrating with Next.js](/how-to/integrating-cycles-with-nextjs) — middleware, server actions, per-tenant isolation
- [Handling Streaming Responses](/how-to/handling-streaming-responses-with-cycles) — streaming patterns in detail
- [Cost Estimation Cheat Sheet](/how-to/cost-estimation-cheat-sheet) — how much to reserve per model
- [Choosing the Right Integration Pattern](/how-to/choosing-the-right-integration-pattern) — when to use `withCycles` vs `reserveForStream`
- [Vercel AI SDK example](https://github.com/runcycles/cycles-client-typescript/tree/main/examples/vercel-ai-sdk) — runnable Vercel AI SDK integration



# Integrations Overview

Cycles has integration patterns for LLM providers, agent frameworks, and web servers. The guides show where to place reserve → execute → commit/release around protected paths. Coverage depends on the hooks and calls each integration actually instruments; uninstrumented traffic is unaffected.

## Supported integrations

| Integration | Language | Streaming | Pattern |
|-------------|----------|-----------|---------|
| **LLM Providers** | | | |
| [OpenAI (Python)](/how-to/integrating-cycles-with-openai) | Python | Yes | Decorator |
| [OpenAI (TypeScript)](/how-to/integrating-cycles-with-openai-typescript) | TypeScript | Yes | `withCycles` / `reserveForStream` |
| [Anthropic (Python)](/how-to/integrating-cycles-with-anthropic) | Python | Yes | Decorator |
| [Anthropic (TypeScript)](/how-to/integrating-cycles-with-anthropic-typescript) | TypeScript | Yes | `withCycles` / `reserveForStream` |
| [AWS Bedrock](/how-to/integrating-cycles-with-aws-bedrock) | TypeScript | Yes | `withCycles` / `reserveForStream` |
| [Google Gemini](/how-to/integrating-cycles-with-google-gemini) | TypeScript | Yes | `withCycles` / `reserveForStream` |
| [Groq](/how-to/integrating-cycles-with-groq) | Python / TypeScript | — | Decorator / `withCycles` |
| [Ollama / Local LLMs](/how-to/integrating-cycles-with-ollama) | Python / TypeScript | — | Decorator / `withCycles` |
| **AI Frameworks** | | | |
| [LangChain](/how-to/integrating-cycles-with-langchain) | Python | Yes | Agent middleware ([`langchain-runcycles`](https://pypi.org/project/langchain-runcycles/)) — `CyclesModelGate` + `CyclesToolGate` + `CyclesFanOutGate` for `create_agent`; callback handler for non-agent runnables |
| [LangChain.js](/how-to/integrating-cycles-with-langchain-js) | TypeScript | Yes | `withCycles` / `reserveForStream` |
| [LangGraph](/how-to/integrating-cycles-with-langgraph) | Python | Yes | Agent middleware ([`langchain-runcycles`](https://pypi.org/project/langchain-runcycles/)) for `create_agent`; managed decorator/reservation for raw `StateGraph` nodes |
| [Vercel AI SDK](/how-to/integrating-cycles-with-vercel-ai-sdk) | TypeScript | Yes | `reserveForStream` |
| [Spring AI](/how-to/integrating-cycles-with-spring-ai) | Java | Yes | `@Cycles` annotation |
| [LlamaIndex](/how-to/integrating-cycles-with-llamaindex) | Python | — | Decorator |
| [CrewAI](/how-to/integrating-cycles-with-crewai) | Python | — | Decorator |
| [Pydantic AI](/how-to/integrating-cycles-with-pydantic-ai) | Python | — | Decorator |
| [AnyAgent](/how-to/integrating-cycles-with-anyagent) | Python | — | Callback (lifecycle hooks) |
| [AutoGen](/how-to/integrating-cycles-with-autogen) | Python | — | Model client wrapper |
| **Agent Platforms** | | | |
| [MCP Server](/how-to/integrating-cycles-with-mcp) | TypeScript (Node.js) | — | MCP tools |
| [OpenAI Agents](/how-to/integrating-cycles-with-openai-agents) | Python | — | RunHooks (lifecycle hooks) |
| [OpenClaw](/how-to/integrating-cycles-with-openclaw) | TypeScript | Yes | Plugin (lifecycle hooks) |
| [AP2 (Agent Payments Protocol)](https://pypi.org/project/runcycles-ap2/) | Python | — | Payment-mandate guard ([`runcycles-ap2`](https://pypi.org/project/runcycles-ap2/)) — reserve / commit / release around AP2 mandates; Cycles idempotency deduplicates accounting, while PSP idempotency or an atomic claim is still required for consume-once execution |
| **Runtime SDKs** | | | |
| [Rust](/how-to/integrating-cycles-with-rust) | Rust | Yes | Tokio async client + RAII guards |
| **Web Frameworks** | | | |
| [Next.js](/how-to/integrating-cycles-with-nextjs) | TypeScript | Yes | `withCycles` / Middleware |
| [Express](/how-to/integrating-cycles-with-express) | TypeScript | Yes | Middleware / `withCycles` |
| [Django](/how-to/integrating-cycles-with-django) | Python | — | Middleware / Decorator |
| [Flask](/how-to/integrating-cycles-with-flask) | Python | — | Decorator / `before_request` |
| [FastAPI](/how-to/integrating-cycles-with-fastapi) | Python | — | Middleware / Decorator |

## Integration patterns

Cycles offers several integration approaches depending on your stack:

### MCP Server

The zero-code tool-exposure approach. Add the Cycles MCP Server to your AI agent's configuration and it discovers `cycles_reserve`, `cycles_commit`, and other budget tools through MCP. This is cooperative, not automatic enforcement of the host's other actions. Hard limits require **Cycles Budget Guard for Claude Code** or a mandatory handler, gateway, harness, or service boundary.

Best for: budget-aware workflows and discovery in Claude Desktop, Claude Code, Cursor, Windsurf, and other MCP-compatible hosts.

### Decorator / Higher-order function

The simplest approach. Wrap your LLM-calling function and Cycles handles reservation, commit, and release automatically.

- **Python:** `@cycles` decorator
- **TypeScript:** `withCycles` higher-order function

Best for: individual model calls, simple request-response flows.

### RunHooks / Lifecycle hooks

For agent frameworks that expose lifecycle hooks. A plugin implements the framework's hook interface to create reservations on start and commit on end — covering the entire agent run automatically.

- **OpenAI Agents SDK:** `CyclesRunHooks` implements the SDK's `RunHooks` interface
- **OpenClaw:** Plugin hooks into `before_model_resolve`, `before_tool_call`, etc.

Best for: multi-agent workflows, tool governance, agent handoff tracking.

### Agent middleware (LangChain 1.x)

For LangChain agents built with `langchain.agents.create_agent`. The [`langchain-runcycles`](https://pypi.org/project/langchain-runcycles/) package provides `AgentMiddleware` subclasses (`CyclesModelGate`, `CyclesToolGate`, `CyclesFanOutGate`) that intercept model calls, tool calls, and model turns *before* execution — denial returns a `ToolMessage` so the agent recovers gracefully, and fan-out can be capped at the model-turn level.

Best for: production LangChain agents, anything using `create_agent`, agent-style LangGraph nodes.

### Callback handler

For framework surfaces that expose only callbacks. Use a lifecycle-managed callback that heartbeats long work and durably records known spend before commit; a raw in-memory `llm_start`/`llm_end` map is not crash-safe.

Best for: bare Python LangChain runnables (`ChatOpenAI` / chains / RAG) when the SDK's managed callback recipe is used. Prefer `@cycles` for raw LangGraph nodes and `withCycles` for LangChain.js.

### `reserveForStream`

For streaming responses where the actual cost is only known after the stream completes. Reserves budget upfront, auto-extends the reservation TTL during streaming, and commits actual usage when the stream finishes.

Best for: streaming chat UIs, Vercel AI SDK, any provider with streaming support.

### Programmatic client

Direct access to the Cycles client for full control over the reservation lifecycle. Use when the higher-level patterns don't fit your architecture.

Best for: custom frameworks, complex orchestration, batch processing.

See [Choosing the Right Integration Pattern](/how-to/choosing-the-right-integration-pattern) for detailed guidance.

## Adding a new integration

All integrations follow the same protocol:

1. **Reserve** budget before the LLM call with an estimated cost
2. **Execute** the model call (respecting any caps returned)
3. **Commit** actual cost from token usage after execution
4. **Release** on error to free held budget

See [Using the Cycles Client Programmatically](/how-to/using-the-cycles-client-programmatically) for the full client API reference.

## Webhook & Observability Integrations

Cycles emits webhook events for budget state changes, reservation denials, tenant lifecycle, and more. Connect to external alerting and incident management systems:

| Integration | Use Case | Guide |
|---|---|---|
| **PagerDuty** | On-call incident response for budget exhaustion and over-limit | [Webhook Integrations](/how-to/webhook-integrations#integration-pagerduty) |
| **Slack** | Channel notifications for budget thresholds and tenant alerts | [Webhook Integrations](/how-to/webhook-integrations#integration-slack) |
| **ServiceNow** | Incident creation for critical budget events | [Webhook Integrations](/how-to/webhook-integrations#integration-servicenow) |
| **Custom receiver** | Direct HTTP endpoint with HMAC verification | [Webhook Integrations](/how-to/webhook-integrations#integration-custom-receiver-direct) |

See [Webhook Integrations](/how-to/webhook-integrations) for full examples with signature verification code in Python, Node.js, and Go.

## Next steps

- [Adding Cycles to an Existing Application](/how-to/adding-cycles-to-an-existing-application) — step-by-step guide for your first integration
- [Webhook Integrations](/how-to/webhook-integrations) — PagerDuty, Slack, ServiceNow webhook examples
- [Cost Estimation Cheat Sheet](/how-to/cost-estimation-cheat-sheet) — pricing reference for estimation
- [Error Handling Patterns](/how-to/error-handling-patterns-in-cycles-client-code) — handling budget errors across languages

## Read the foundations

For the layer-by-layer view of where these integrations sit relative to other agent control approaches — wrappers, provider-client patches, framework hooks, LLM gateways, observability — and why runtime authority complements them rather than replaces them:

- [Python AI Agent Control: Cost, Risk, and Audit by Layer](/blog/python-ai-agent-control-cost-risk-audit-layers) — six layers walked through, what each covers across cost / risk / audit, and where each stops short.
- [How Cycles Meters Caller-Assigned Action Exposure](/blog/beyond-budget-how-cycles-controls-agent-actions) — how applications combine tool authorization with caller-assigned `RISK_POINTS` at instrumented boundaries.
- [Why Local-First Agent Runtimes Need Runtime Authority](/blog/every-local-first-agent-runtime-needs-budget-authority) — local-first / BYOK category context for OpenClaw, Cline, Aider, Continue, and similar runtimes.
- [Agents Are Cross-Cutting. Your Controls Aren't.](/blog/agents-are-cross-cutting-your-controls-arent) — the structural argument for why agent governance has to span every integration the agent uses.

## Related concepts

- [Tracking tokens in a streaming LLM response](/blog/tracking-tokens-in-a-streaming-llm-response)
- [What is runtime authority?](/blog/what-is-runtime-authority-for-ai-agents)
- [AI agent action control: hard limits on side effects](/blog/ai-agent-action-control-hard-limits-side-effects)



# Managing Webhooks

This guide covers the full webhook lifecycle: creating subscriptions, testing connectivity, monitoring delivery health, handling failures, rotating secrets, and replaying events.

::: tip Webhook operations from the dashboard
Every action in this guide — create, test, replay, pause/enable, reset failures, delete — is also available on the Webhooks page in the [Cycles Admin Dashboard](/quickstart/deploying-the-cycles-dashboard). The dashboard shows subscription health (green/yellow/red), recent delivery history, and supports **bulk pause / enable** with tenant filtering. Use the dashboard for day-two operations and the curl examples below for automation.
:::

## Creating a Webhook Subscription

### Admin subscription

Required fields: `url` and `event_types` (at least one event type on create). Add `?tenant_id=acme-corp` to scope the subscription to a specific tenant; omit for system-wide subscriptions (all tenants). All other fields are optional — the server provides sensible defaults (`signing_secret` is auto-generated if omitted). On update (`PATCH`), `event_types` may be cleared to empty as long as `event_categories` is non-empty (a category-only subscription); the server rejects only the empty-both state.

```bash
# Tenant-scoped subscription (receives events for acme-corp only)
curl -X POST 'http://localhost:7979/v1/admin/webhooks?tenant_id=acme-corp' \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-endpoint.example.com/cycles-webhook",
    "event_types": ["budget.exhausted", "budget.over_limit_entered", "reservation.denied"],
    "retry_policy": {
      "max_retries": 5,
      "initial_delay_ms": 1000,
      "backoff_multiplier": 2.0,
      "max_delay_ms": 60000
    },
    "disable_after_failures": 10
  }'
```

The response includes the `subscription_id` and `signing_secret`. **Store the signing secret securely** — it's returned only once.

```json
{
  "subscription": {
    "subscription_id": "whsub_abc123...",
    "status": "ACTIVE",
    "consecutive_failures": 0,
    ...
  },
  "signing_secret": "your-secret-here"
}
```

### Auto-generated signing secret

If you omit `signing_secret`, the server generates a cryptographically random one:

```bash
curl -X POST http://localhost:7979/v1/admin/webhooks \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-endpoint.example.com/webhook",
    "event_types": ["budget.exhausted"]
  }'
```

The generated secret (e.g., `whsec_dGVzdC1zZWNy...`) is in the response. Copy it immediately.

### Category-based subscriptions

Subscribe to **all events in a category** using `event_categories`. This is additive with `event_types` — if you specify both, you get the union. Note: on **create**, `event_types` must be non-empty, so include a representative type alongside the category wildcard. (A later `PATCH` may clear `event_types` to leave a category-only subscription; the server rejects only the state where both arrays are empty.)

```bash
# All budget events (17 types) + all reservation events (6 types)
curl -X POST http://localhost:7979/v1/admin/webhooks \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-endpoint.example.com/webhook",
    "event_types": ["budget.created"],
    "event_categories": ["budget", "reservation"]
  }'
```

> **Note:** Category subscriptions receive future event types added to that category in new releases, without subscription changes.

### Scope filtering

Narrow events to specific scopes:

```bash
# Only events for the prod workspace
curl -X POST http://localhost:7979/v1/admin/webhooks \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-endpoint.example.com/webhook",
    "event_types": ["budget.exhausted"],
    "scope_filter": "tenant:acme-corp/workspace:prod/*"
  }'
```

### Tenant-scoped subscriptions

Subscribe to events for a specific tenant by passing `tenant_id` as a query parameter:

```bash
curl -X POST "http://localhost:7979/v1/admin/webhooks?tenant_id=acme-corp" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://acme-corp.example.com/webhook",
    "event_types": ["budget.exhausted", "reservation.denied"]
  }'
```

Omit `tenant_id` for system-wide subscriptions (receives events from all tenants).

## Testing a Webhook

Before relying on a webhook, verify connectivity:

```bash
curl -X POST http://localhost:7979/v1/admin/webhooks/whsub_abc123/test \
  -H "X-Admin-API-Key: $ADMIN_KEY"
```

Response:

```json
{
  "success": true,
  "response_status": 200,
  "response_time_ms": 42,
  "event_id": "evt_test_abc123"
}
```

The test sends a `system.webhook_test` event to the subscription's URL. It does **not** count toward consecutive failures or affect subscription status.

## Listing Subscriptions

```bash
# All subscriptions
curl http://localhost:7979/v1/admin/webhooks \
  -H "X-Admin-API-Key: $ADMIN_KEY"

# Filter by status
curl "http://localhost:7979/v1/admin/webhooks?status=DISABLED" \
  -H "X-Admin-API-Key: $ADMIN_KEY"

# Filter by tenant
curl "http://localhost:7979/v1/admin/webhooks?tenant_id=acme-corp" \
  -H "X-Admin-API-Key: $ADMIN_KEY"
```

## Monitoring Delivery Health

### Check delivery history

```bash
curl "http://localhost:7979/v1/admin/webhooks/whsub_abc123/deliveries?status=FAILED&limit=10" \
  -H "X-Admin-API-Key: $ADMIN_KEY"
```

Response shows delivery attempts with status, response code, and error details:

```json
{
  "deliveries": [
    {
      "delivery_id": "del_xyz789",
      "event_id": "evt_abc123",
      "event_type": "budget.exhausted",
      "status": "FAILED",
      "attempts": 6,
      "response_status": 503,
      "error_message": "HTTP 503",
      "attempted_at": "2026-04-01T12:00:00Z",
      "completed_at": "2026-04-01T12:05:32Z"
    }
  ],
  "has_more": false
}
```

### Check subscription health

```bash
curl http://localhost:7979/v1/admin/webhooks/whsub_abc123 \
  -H "X-Admin-API-Key: $ADMIN_KEY"
```

Key fields to monitor:
- `consecutive_failures` — number of deliveries that failed in a row (resets to 0 on any success)
- `status` — `ACTIVE`, `PAUSED`, or `DISABLED`
- `last_success_at` — when the last delivery succeeded
- `last_failure_at` — when the last delivery failed

### Redis queue depth

```bash
# Pending deliveries (waiting for events service to process)
redis-cli LLEN dispatch:pending

# Deliveries in retry queue
redis-cli ZCARD dispatch:retry
```

If `dispatch:pending` grows continuously, the events service may be down or overwhelmed.

### Prometheus metrics (v0.1.25.6+)

The events service publishes webhook delivery metrics on `/actuator/prometheus` (management port `9980`, which is unauthenticated in the reference deployment — restrict it at the network layer) under the `cycles_webhook_*` namespace. The operationally most useful alerts:

- **`cycles_webhook_subscription_auto_disabled_total`** — any increase means a receiver has gone from healthy to dead. Page on `rate(cycles_webhook_subscription_auto_disabled_total[5m]) > 0`.
- **`cycles_webhook_delivery_failed_total`** — failed delivery attempts, tagged by `reason`. The reason values are `event_not_found`, `subscription_not_found`, `subscription_inactive`, `http_4xx`, `http_5xx`, `transport_error`, and `ssrf_blocked`. Spikes in `http_5xx` or `transport_error` (connect/read timeouts, DNS failures) signal either a widespread receiver outage or a configuration regression.
- **`cycles_webhook_delivery_stale_total`** — non-zero means the `MAX_DELIVERY_AGE_MS` gate (default 24h) is firing. Usually benign after an events-service outage; persistently firing means dispatch is not catching up.
- **`cycles_webhook_delivery_latency_seconds`** — Timer with `outcome` tag. Percentile histograms are not enabled by default, so no `_bucket` series exist and `histogram_quantile()` won't work out of the box — watch `cycles_webhook_delivery_latency_seconds_max` and the `_sum`/`_count` average instead. A creeping `_max` is often the first signal that a receiver is degrading before it starts outright failing. (To get true percentiles, enable percentile histograms for this timer via Micrometer configuration.)

See [Deploying the Events Service](/quickstart/deploying-the-events-service#prometheus-metrics) for the full metric inventory.

## Handling Failures

### Subscription statuses

| Status | Meaning | Deliveries | How to fix |
|---|---|---|---|
| `ACTIVE` | Normal operation | Delivering | — |
| `PAUSED` | Manually paused | **Not queued** — events emitted during the pause are dropped for this subscription, not held for later | `PATCH` status to `ACTIVE`, then [replay](#replaying-events) the pause window to backfill |
| `DISABLED` | Auto-disabled after consecutive failures | **Not queued** | Fix endpoint, then `PATCH` status to `ACTIVE`; replay to backfill |

### Re-enabling a disabled subscription

When a subscription is auto-disabled (e.g., 10 consecutive failures), fix the underlying issue first, then:

```bash
curl -X PATCH http://localhost:7979/v1/admin/webhooks/whsub_abc123 \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"status": "ACTIVE"}'
```

This resets `consecutive_failures` to 0 and resumes delivery.

### Pausing and resuming

```bash
# Pause (e.g., during maintenance)
curl -X PATCH http://localhost:7979/v1/admin/webhooks/whsub_abc123 \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"status": "PAUSED"}'

# Resume
curl -X PATCH http://localhost:7979/v1/admin/webhooks/whsub_abc123 \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"status": "ACTIVE"}'
```

## Updating a Subscription

Partial update — only provided fields change:

```bash
# Change URL
curl -X PATCH http://localhost:7979/v1/admin/webhooks/whsub_abc123 \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://new-endpoint.example.com/webhook"}'

# Change event types (replaces, does not merge)
curl -X PATCH http://localhost:7979/v1/admin/webhooks/whsub_abc123 \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"event_types": ["budget.exhausted", "reservation.commit_overage", "reservation.denied"]}'

# Switch to a category-only subscription: clear event_types, keep categories.
# Valid on update (unlike create); the server rejects only the empty-both state.
curl -X PATCH http://localhost:7979/v1/admin/webhooks/whsub_abc123 \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"event_types": [], "event_categories": ["budget", "reservation"]}'

# Adjust retry policy
curl -X PATCH http://localhost:7979/v1/admin/webhooks/whsub_abc123 \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"retry_policy": {"max_retries": 10, "max_delay_ms": 120000}}'
```

## Rotating Signing Secrets

To rotate the HMAC signing secret:

```bash
curl -X PATCH http://localhost:7979/v1/admin/webhooks/whsub_abc123 \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"signing_secret": "new-secret-value"}'
```

**Rotation procedure:**
1. Generate new secret
2. Update the subscription with the new secret
3. Update the receiver to accept both old and new signatures (dual verification)
4. Once all in-flight retries with the old secret complete, remove old secret from receiver

## Replaying Events

Re-deliver historical events to a subscription (e.g., after fixing a broken endpoint):

```bash
curl -X POST http://localhost:7979/v1/admin/webhooks/whsub_abc123/replay \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "from": "2026-04-01T00:00:00Z",
    "to": "2026-04-01T23:59:59Z",
    "max_events": 100
  }'
```

Response:

```json
{
  "replay_id": "replay_abc123",
  "events_queued": 47,
  "estimated_completion_seconds": 5
}
```

Filter by event type:

```bash
curl -X POST http://localhost:7979/v1/admin/webhooks/whsub_abc123/replay \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "from": "2026-04-01T00:00:00Z",
    "to": "2026-04-01T23:59:59Z",
    "event_types": ["budget.exhausted"],
    "max_events": 1000
  }'
```

## Deleting a Subscription

```bash
curl -X DELETE http://localhost:7979/v1/admin/webhooks/whsub_abc123 \
  -H "X-Admin-API-Key: $ADMIN_KEY"
```

Returns `204 No Content`. Deletion is irreversible, and pending deliveries for the subscription are cancelled.

## Querying Events

Browse the event stream independent of webhooks:

```bash
# All events for a tenant
curl "http://localhost:7979/v1/admin/events?tenant_id=acme-corp&limit=20" \
  -H "X-Admin-API-Key: $ADMIN_KEY"

# Filter by type and time range
curl "http://localhost:7979/v1/admin/events?event_type=budget.exhausted&from=2026-04-01T00:00:00Z&to=2026-04-02T00:00:00Z" \
  -H "X-Admin-API-Key: $ADMIN_KEY"

# Get a single event by ID
curl http://localhost:7979/v1/admin/events/evt_abc123 \
  -H "X-Admin-API-Key: $ADMIN_KEY"
```

## Tenant Self-Service

Tenants manage their own webhooks via `/v1/webhooks` using `X-Cycles-API-Key`. The tenant is derived from the key; do not pass a tenant query parameter with tenant-scoped auth.

A tenant-owned subscription is restricted to **tenant-accessible** event classes — `budget.*`, `reservation.*`, `tenant.*` — for both `event_types` and `event_categories`; admin-only classes (`api_key.*`, `policy.*`, `webhook.*`, `system.*`) are rejected with `400 INVALID_REQUEST` (governance WEBHOOK SUBSCRIPTION INVARIANT 2). The same rule binds a tenant-owned row created via the admin plane (`POST /v1/admin/webhooks?tenant_id=X`) or admin-on-behalf-of — it is a property of the owning tenant, not the caller. To monitor a specific tenant's admin-only events, create a `__system__`-owned subscription (no `tenant_id`) and filter client-side on the envelope `tenant_id` — see [Tenant-accessible events](/protocol/webhook-event-delivery-protocol#tenant-accessible-events).

```bash
# Create (restricted to budget.*, reservation.*, tenant.* events)
curl -X POST http://localhost:7979/v1/webhooks \
  -H "X-Cycles-API-Key: $TENANT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://acme.example.com/budget-alerts",
    "event_types": ["budget.exhausted", "reservation.denied"]
  }'

# List tenant's subscriptions
curl http://localhost:7979/v1/webhooks \
  -H "X-Cycles-API-Key: $TENANT_API_KEY"

# Get one subscription
curl http://localhost:7979/v1/webhooks/whsub_abc123 \
  -H "X-Cycles-API-Key: $TENANT_API_KEY"

# Pause or resume delivery
curl -X PATCH http://localhost:7979/v1/webhooks/whsub_abc123 \
  -H "X-Cycles-API-Key: $TENANT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"status": "PAUSED"}'

# Send a test delivery to the subscription URL
curl -X POST http://localhost:7979/v1/webhooks/whsub_abc123/test \
  -H "X-Cycles-API-Key: $TENANT_API_KEY"

# Inspect delivery attempts
curl "http://localhost:7979/v1/webhooks/whsub_abc123/deliveries?status=FAILED&limit=10" \
  -H "X-Cycles-API-Key: $TENANT_API_KEY"

# Delete a subscription
curl -X DELETE http://localhost:7979/v1/webhooks/whsub_abc123 \
  -H "X-Cycles-API-Key: $TENANT_API_KEY"

# Query tenant's events
curl "http://localhost:7979/v1/events?event_type=budget.exhausted" \
  -H "X-Cycles-API-Key: $TENANT_API_KEY"
```

**Required permissions:** `webhooks:write` (create/update/delete/test), `webhooks:read` (list/get delivery history), `events:read` (query events). These are not included in default key permissions — they must be explicitly requested at key creation. See [API Key Permissions](/how-to/api-key-management-in-cycles#available-permissions-27-total) for the full list.

## Webhook URL Security

The events service always applies a delivery-time SSRF baseline unless its development-only escape hatch is enabled. It rejects `0.0.0.0/8`, `10.0.0.0/8`, `100.64.0.0/10`, `127.0.0.0/8`, `169.254.0.0/16`, `172.16.0.0/12`, `192.168.0.0/16`, `::1/128`, `fe80::/10`, `fc00::/7`, and any-local or unspecified addresses. Admin-configured CIDR blocks are additive; `allowed_url_patterns` only narrows accepted targets and cannot bypass the baseline.

To view and narrow the admin-side policy:

```bash
# View current security config
curl http://localhost:7979/v1/admin/config/webhook-security \
  -H "X-Admin-API-Key: $ADMIN_KEY"

# Restrict production delivery to an approved public endpoint
curl -X PUT http://localhost:7979/v1/admin/config/webhook-security \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "allowed_url_patterns": ["https://hooks.example.com/cycles/*"],
    "blocked_cidr_ranges": ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]
  }'

# Enable HTTP at the admin boundary for development/testing
curl -X PUT http://localhost:7979/v1/admin/config/webhook-security \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"allow_http": true, "blocked_cidr_ranges": []}'
```

Local or private-network delivery also requires
`WEBHOOK_URL_GUARD_ALLOW_PRIVATE_NETWORKS=true` on the events service and an events-service restart. Both that environment variable and `allow_http: true` are required for a private HTTP target. Never enable the private-network escape hatch in production.

## Next Steps

- [Webhook Integrations](/how-to/webhook-integrations) — PagerDuty, Slack, ServiceNow examples with signature verification
- [Webhooks and Events Concepts](/concepts/webhooks-and-events) — architecture, delivery semantics, event types
- [Security Hardening](/how-to/security-hardening) — encryption, SSRF, secret rotation
- [Production Operations](/how-to/production-operations-guide) — events service deployment and failure handling



# Migrating from a Custom Rate Limiter to Cycles

If you've built a custom rate limiter for your AI agents — Redis counters, per-provider spend tracking, manual cost tables — and you're hitting the walls described in [We Built a Custom Agent Rate Limiter. Here's Why We Stopped](/blog/we-built-a-custom-agent-rate-limiter-heres-why-we-stopped), this guide walks you through replacing it with Cycles.

The migration is **zero-risk** because Cycles runs in [shadow mode](/how-to/shadow-mode-in-cycles-how-to-roll-out-budget-enforcement-without-breaking-production) alongside your existing limiter. You validate before you cut over. If anything looks wrong, your old limiter is still enforcing.

**Timeline:** 4 weeks (1 week setup, 2 weeks shadow mode, 1 week cut-over + cleanup)

## Before you start

**You'll need:**
- Docker (for Cycles server + Redis)
- Your current rate limiter code (to compare behavior)
- An admin API key (created during setup)
- ~30 minutes for initial deployment

**Your existing limiter keeps running** throughout phases 1-3. You only disable it in phase 4 after Cycles is validated.

## Phase 1: Deploy Cycles alongside your existing limiter (days 1-2)

### Start the server

```bash
docker compose -f docker-compose.full-stack.yml up -d
```

See the [full deployment guide](/quickstart/deploying-the-full-cycles-stack) for details. The Cycles server runs on port 7878, the admin server on port 7979.

### Create your first tenant and API key

```bash
# Create a tenant matching your current user/org concept
curl -X POST http://localhost:7979/v1/admin/tenants \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"tenant_id": "acme-corp", "name": "Acme Corp"}'

# Create an API key for your application
curl -X POST http://localhost:7979/v1/admin/api-keys \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "tenant_id": "acme-corp",
    "name": "app-server",
    "permissions": ["reservations:create", "reservations:commit", "reservations:release", "reservations:extend", "reservations:list", "balances:read", "budgets:write"]
  }'
# Save the returned key_secret value — it won't be shown again
```

### Create a budget matching your current cap

Map your existing rate limit to a Cycles budget:

```bash
# If your current cap is $50/month per user (1 USD = 100,000,000 microcents)
curl -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "scope": "tenant:acme-corp",
    "unit": "USD_MICROCENTS",
    "allocated": { "amount": 5000000000, "unit": "USD_MICROCENTS" }
  }'
```

## Phase 2: Map your current system to Cycles

Before writing code, translate your existing concepts:

| Your custom rate limiter | Cycles equivalent |
|---|---|
| Redis counter per user | [Budget scope](/how-to/understanding-tenants-scopes-and-budgets-in-cycles) (per-tenant) |
| `GET balance → check → INCRBY` | [Reserve](/glossary#reservation) (atomic, no TOCTOU) |
| Report actual cost after call | [Commit](/glossary#commit) (reconciles estimate vs actual) |
| Cancel a pending operation | [Release](/glossary#release) (returns held budget) |
| Per-provider spend tracking | Multi-scope budgets (one per provider, or single aggregate) |
| Manual cost estimate table | `estimate` field on reservation |
| "Overspend by 10% OK" policy | `ALLOW_WITH_OVERDRAFT` [overage policy](/how-to/choosing-the-right-overage-policy) |
| Hard deny at cap | `REJECT` overage policy |
| Per-user monthly cap | Tenant budget with periodic reset via admin API |
| Alert on threshold | [Webhook events](/protocol/webhook-event-delivery-protocol) (`budget.exhausted`) |
| No action-level control | [RISK_POINTS](/how-to/assigning-risk-points-to-agent-tools) budgets |
| No retry deduplication | [Idempotency keys](/glossary#idempotency-key) on every operation |

## Phase 3: Dual-write in shadow mode (weeks 1-2)

This is the critical phase. Your existing limiter keeps enforcing. Cycles runs alongside in shadow mode, logging what it *would* do.

### Install the Python client

```bash
pip install runcycles
```

### Add Cycles calls next to your existing limiter

**Before (your custom limiter only):**

```python
import redis

r = redis.Redis()

def check_and_charge(user_id, estimated_cost):
    balance = int(r.get(f"budget:{user_id}") or 0)
    cap = int(r.get(f"cap:{user_id}") or 50_000_000)
    if balance + estimated_cost > cap:
        raise Exception("Budget exceeded")
    # ... do the LLM call ...
    actual_cost = get_actual_cost()
    r.incrby(f"budget:{user_id}", actual_cost)
```

**After (dual-write with Cycles in shadow mode):**

```python
import redis
from runcycles import (
    CyclesConfig, CyclesClient, DecisionRequest,
    Subject, Action, Amount, Unit
)
import uuid

r = redis.Redis()

config = CyclesConfig(
    base_url="http://localhost:7878",
    api_key="cyc_live_...",
    tenant="acme-corp",
)
client = CyclesClient(config)

def check_and_charge(user_id, estimated_cost):
    # Your existing limiter still enforces
    balance = int(r.get(f"budget:{user_id}") or 0)
    cap = int(r.get(f"cap:{user_id}") or 50_000_000)
    if balance + estimated_cost > cap:
        raise Exception("Budget exceeded")

    # Cycles shadow check — decide() evaluates without creating a reservation
    try:
        response = client.decide(DecisionRequest(
            idempotency_key=str(uuid.uuid4()),
            subject=Subject(tenant="acme-corp", workspace="production", agent=user_id),
            action=Action(kind="llm.completion", name="gpt-4o"),
            estimate=Amount(unit=Unit.USD_MICROCENTS, amount=estimated_cost),
        ))
        # Log: would Cycles have allowed or denied this?
        decision = response.get_body_attribute("decision")
        print(f"Cycles decision: {decision} for {user_id}")
    except Exception as e:
        # Shadow check failure should never block your app
        print(f"Cycles shadow error (non-blocking): {e}")

    # ... do the LLM call ...
    actual_cost = get_actual_cost()

    # Your existing limiter records actual spend
    r.incrby(f"budget:{user_id}", actual_cost)
```

### What to watch during shadow mode

Run for **1-2 weeks** and compare:

| Metric | What to check |
|---|---|
| **Agreement rate** | How often does Cycles agree with your limiter? (should be >95%) |
| **False denials** | Did Cycles deny something your limiter allowed? (indicates budget too tight) |
| **Missed denials** | Did your limiter deny something Cycles would have allowed? (indicates your limiter is tighter) |
| **Decision latency** | How much time does the Cycles call add? (expect ~5ms p50 for decide) |

If agreement is <90%, your Cycles budget needs adjusting before cut-over.

## Phase 4: Cut over (week 3)

When shadow mode looks good (>95% agreement, no surprises), switch Cycles from shadow to enforcement:

### Step 1: Replace decide() with create_reservation()

```python
# Switch from shadow decide() to enforcing create_reservation()
from runcycles import (
    ReservationCreateRequest, CommitRequest,
    Subject, Action, Amount, Unit
)

response = client.create_reservation(ReservationCreateRequest(
    idempotency_key=idempotency_key,
    subject=Subject(tenant="acme-corp", workspace="production", agent=user_id),
    action=Action(kind="llm.completion", name="gpt-4o"),
    estimate=Amount(unit=Unit.USD_MICROCENTS, amount=estimated_cost),
))

# Now check the decision — Cycles is enforcing
if not response.is_success:
    raise Exception("Budget exceeded")

reservation_id = response.get_body_attribute("reservation_id")
```

### Step 2: Add commit after work completes

```python
# After the LLM call, commit actual cost
client.commit_reservation(reservation_id, CommitRequest(
    idempotency_key=f"commit-{idempotency_key}",
    actual=Amount(unit=Unit.USD_MICROCENTS, amount=actual_cost),
))
```

### Step 3: Disable old limiter

```python
# Uses imports and client setup from Phase 3 + Phase 4 Step 1
import uuid

def check_and_charge(user_id, estimated_cost):
    # OLD LIMITER — disabled, kept as comment for rollback
    # balance = int(r.get(f"budget:{user_id}") or 0)
    # cap = int(r.get(f"cap:{user_id}") or 50_000_000)
    # if balance + estimated_cost > cap:
    #     raise Exception("Budget exceeded")

    # Cycles enforcing
    idempotency_key = str(uuid.uuid4())
    response = client.create_reservation(ReservationCreateRequest(
        idempotency_key=idempotency_key,
        subject=Subject(tenant="acme-corp", workspace="production", agent=user_id),
        action=Action(kind="llm.completion", name="gpt-4o"),
        estimate=Amount(unit=Unit.USD_MICROCENTS, amount=estimated_cost),
    ))

    if not response.is_success:
        raise Exception("Budget exceeded")

    reservation_id = response.get_body_attribute("reservation_id")

    # ... do the LLM call ...
    actual_cost = get_actual_cost()

    client.commit_reservation(reservation_id, CommitRequest(
        idempotency_key=f"commit-{idempotency_key}",
        actual=Amount(unit=Unit.USD_MICROCENTS, amount=actual_cost),
    ))
```

### Or use the decorator (cleaner)

```python
import openai
from runcycles import cycles, set_default_client

set_default_client(client)

@cycles(
    estimate=lambda prompt, max_tokens: max_tokens * 10,
    actual=lambda result: len(result) * 5,
    action_kind="llm.completion",
    action_name="gpt-4o",
    workspace="production",
)
def call_llm(prompt: str, max_tokens: int) -> str:
    return openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
    ).choices[0].message.content
```

The decorator handles reserve, commit, release (on failure), and idempotency automatically.

## Phase 5: Cleanup (week 4)

- Remove commented-out limiter code
- Remove old Redis keys (`DEL budget:* cap:*`)
- Consider adding [RISK_POINTS](/how-to/assigning-risk-points-to-agent-tools) budget for action control
- Set up [webhook integrations](/how-to/webhook-integrations) (PagerDuty, Slack)
- Review [common budget patterns](/how-to/common-budget-patterns) for multi-tenant, per-workflow structures

## Rollback plan

At **any phase**, you can revert to your old limiter:

1. Re-enable old limiter code (uncomment)
2. Switch Cycles calls back to `client.decide()` (shadow mode) or remove them entirely
3. No data migration needed — both systems track state independently

The migration is safe because:
- Phase 1-3: your old limiter is still enforcing
- Phase 4: you can re-enable the old limiter in minutes
- The two systems share no state — reverting Cycles does not affect your Redis counters

## What you gain after migration

| Capability | Custom limiter | After migration (Cycles) |
|---|---|---|
| Atomic budget check | No (TOCTOU race) | Yes (atomic Lua script) |
| Cross-provider budget | Manual per-provider tracking | Single scope hierarchy |
| Retry deduplication | No | Idempotency keys on every operation |
| Action-level risk control | No | RISK_POINTS budgets |
| Webhook alerts | Custom implementation | Built-in (51 registered event types across 7 categories, PagerDuty/Slack) |
| Multi-tenant isolation | Manual Redis key prefixing | Built-in scope hierarchy |
| Delegation attenuation | No | Explicit child ledgers plus orchestrator-owned tool/depth restrictions |
| Shadow mode validation | No | `decide()` endpoint for shadow evaluation |
| Graceful degradation | No | ALLOW_WITH_CAPS with tool denylists |

## Common migration questions

**Can I migrate one agent at a time?**
Yes. Each agent can have its own subject scope. Migrate `agent:support-bot` first, then `agent:sales-bot`, etc. Unmigrated agents keep using the old limiter.

**What if my budget periods don't match?**
Use the admin API to reset budgets on your schedule: `POST /v1/admin/budgets/fund?scope={scope}&unit={unit}` with a `RESET_SPENT` operation (clears `spent` for the new period). This can be triggered from a cron job. For migrations where you need to import a customer mid-period with existing consumption, `RESET_SPENT` also accepts an optional `spent` field to set the starting consumption explicitly.

**Do I need to migrate all providers at once?**
No. You can start with one provider (e.g., OpenAI) and add others incrementally. Each reservation specifies the provider via the `action` field.

**What about historical spend data?**
Cycles starts fresh. Keep your old Redis data for comparison during shadow mode. After migration is complete, the old data can be archived or deleted.

**What if Cycles server goes down?**
Your application code should handle reservation failures gracefully. If the reserve call fails, decide whether to fail-safe (block the action) or fail-open (allow the action without enforcement). See [degradation paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) for patterns.

## Next steps

- [Full Stack Deployment Guide](/quickstart/deploying-the-full-cycles-stack) — detailed server setup
- [Python Client Quickstart](/quickstart/getting-started-with-the-python-client) — SDK reference
- [Common Budget Patterns](/how-to/common-budget-patterns) — per-tenant, per-workflow, per-run structures
- [Assigning RISK_POINTS to Tools](/how-to/assigning-risk-points-to-agent-tools) — add action authority after cost control
- [Shadow Mode Rollout](/how-to/shadow-mode-in-cycles-how-to-roll-out-budget-enforcement-without-breaking-production) — detailed shadow mode guide

## Related concepts

- [AI agent action control: hard limits on side effects](/blog/ai-agent-action-control-hard-limits-side-effects)
- [Agent delegation chains and authority attenuation](/blog/agent-delegation-chains-authority-attenuation-not-trust-propagation)
- [Graceful degradation patterns](/blog/when-budget-runs-out-graceful-degradation-patterns-for-ai-agents)



# Monitoring and Alerting

This guide covers key metrics to monitor, alerting thresholds, and observability patterns for a production Cycles deployment.

## Key metrics

### Budget utilization

The most important metric. Track the ratio of spent to allocated for each scope:

```
utilization = (spent + reserved) / allocated × 100%
```

**Alert thresholds:**

| Level | Threshold | Action |
|---|---|---|
| Warning | 80% | Notify team. Budget is running low — consider funding or reducing usage. |
| Critical | 95% | Page on-call. Imminent budget exhaustion will start denying requests. |
| Exhausted | 100% | All reservations denied. Fund immediately or accept denial. |

### Query balances for monitoring

```bash
# Get all balances for a tenant
curl -s "http://localhost:7878/v1/balances?tenant=acme-corp" \
  -H "X-Cycles-API-Key: $API_KEY" | jq '.balances[] | {scope, allocated, remaining, spent, reserved, debt}'
```

Build a polling monitor that queries balances and pushes to your metrics system:

```python
import time
import requests

def poll_budgets():
    response = requests.get(
        "http://localhost:7878/v1/balances",
        params={"tenant": "acme-corp"},
        headers={"X-Cycles-API-Key": API_KEY},
    )
    for balance in response.json()["balances"]:
        allocated = balance["allocated"]["amount"]
        if allocated > 0:
            utilization = (balance["spent"]["amount"] + balance["reserved"]["amount"]) / allocated
            push_metric(
                name="cycles.budget.utilization",
                value=utilization,
                tags={"scope": balance["scope"], "unit": balance["allocated"]["unit"]},
            )
            push_metric(
                name="cycles.budget.remaining",
                value=balance["remaining"]["amount"],
                tags={"scope": balance["scope"]},
            )

while True:
    poll_budgets()
    time.sleep(60)  # Poll every minute
```

For the full list of fields available on every reservation and event, see [Standard Metrics and Metadata](/protocol/standard-metrics-and-metadata-in-cycles).

### Reservation metrics

Track reservation lifecycle events:

| Metric | What to watch |
|---|---|
| **Reservations created/sec** | Throughput baseline. Sudden spikes may indicate loops. |
| **Reservation denial rate** | Percentage of reservations denied (`BUDGET_EXCEEDED`). High rates mean budgets are too tight or traffic is too high. |
| **Reservation TTL expiry rate** | Reservations expiring before commit. Indicates operations are taking too long or heartbeat is not working. |
| **Average reservation duration** | Time from reserve to commit. Growing duration may indicate slow downstream services. |
| **Active reservation count** | Current in-flight reservations. Sustained growth suggests commit/release failures. |

### Client recovery signals

The official Python, TypeScript, Spring Boot, and Rust lifecycle helpers durably journal known-actual settlement. Monitor the application processes that run those SDKs, not only the Cycles server:

| Signal | Why it matters |
|---|---|
| Journal write or permission failure | The synchronous settlement may still succeed, but restart recovery is not guaranteed for a record that could not be persisted. |
| Quarantined record | A malformed or unsupported journal record was isolated; other records continue replaying, but this one needs investigation. |
| Retry exhaustion or retained authentication failure | Actual usage is known and safely retained, but the ledger has not converged yet. |
| Expired-commit event fallback failure | The reservation can no longer be committed; the durable event settlement is still pending. |
| Heartbeat stop disposition | The guarded operation continues, but its lease may expire before final settlement. |
| Oldest pending record age and pending count | Sustained growth indicates a server, credential, filesystem, or replay-worker problem. |

Put `~/.runcycles/commit-journal`—or the configured journal directory—on persistent storage and collect SDK warnings from application logs. A graceful shutdown should invoke or allow the SDK's bounded drain; a timed-out drain must leave records intact for the next start.

### Server health metrics

All three Cycles services expose Spring Boot Actuator. The exposed endpoints are `health`, `info`, and `prometheus`. On the runtime and admin servers (since `cycles-server` 0.1.25.45 and the matching admin release), the aggregate `/actuator/health` and `/actuator/prometheus` endpoints require the `X-Admin-API-Key` header; only the liveness/readiness probe sub-paths stay public for orchestrators:

```bash
# Cycles Server (runtime) — aggregate health + prometheus need the admin key
curl -H "X-Admin-API-Key: $ADMIN_KEY" http://localhost:7878/actuator/health
curl -H "X-Admin-API-Key: $ADMIN_KEY" http://localhost:7878/actuator/prometheus
curl http://localhost:7878/actuator/health/liveness    # public
curl http://localhost:7878/actuator/health/readiness   # public

# Admin Server — same auth model
curl -H "X-Admin-API-Key: $ADMIN_KEY" http://localhost:7979/actuator/health
curl http://localhost:7979/actuator/health/liveness    # public
curl http://localhost:7979/actuator/health/readiness   # public
curl -H "X-Admin-API-Key: $ADMIN_KEY" http://localhost:7979/actuator/prometheus

# Events Service — management port 9980, no auth filter in the reference
# deployment; restrict it at the network layer
curl http://localhost:9980/actuator/health
curl http://localhost:9980/actuator/prometheus
```

::: tip Liveness/readiness probes
All three services enable Spring's liveness/readiness probes (`management.endpoint.health.probes.enabled=true`), exposing `/actuator/health/liveness` and `/actuator/health/readiness`. The readiness group includes the Redis health indicator. These probe paths are deliberately exempt from the admin-key requirement so Kubernetes can call them.
:::

Key server metrics (all derived from Spring Boot's default Micrometer registrations — see [Observability Setup](/how-to/observability-setup) for the full metric list):

| Metric | Component | Threshold |
|---|---|---|
| Response latency — `http_server_requests_seconds_max` (or p99 from `_bucket` if you enable percentile histograms) | Cycles Server | Alert if > 50ms |
| Error rate (5xx) — `http_server_requests_seconds_count{status=~"5.."}` | Cycles Server, Admin Server | Alert if > 1% |
| JVM heap usage — `jvm_memory_used_bytes{area="heap"}` / `jvm_memory_max_bytes{area="heap"}` | All services | Alert if > 80% |
| Redis connection pool usage | All services | No server-side metric exposed today — monitor via Redis `CLIENT LIST` or a Redis exporter. |

### Events Service metrics

The Events Service delivers webhooks asynchronously. Its management port is 9980 by default; the app port 7980 has no operator-facing API in the current reference service. Monitor separately:

| Metric | What to watch |
|---|---|
| **Queue depth** (`redis-cli LLEN dispatch:pending`) | Sustained growth means delivery is falling behind. Should be near zero. |
| **Delivery success rate** | Percentage of deliveries receiving HTTP 2xx. Drops indicate endpoint issues. |
| **Retry rate** | High retry rates signal unreliable webhook endpoints or network issues. |
| **Auto-disabled subscriptions** | Any auto-disabled subscription needs investigation — the endpoint failed repeatedly. |
| **Delivery latency** | Time from event creation to successful delivery. Growing latency signals backlog. |

## Alerting rules

::: info Custom `cycles_*` metrics ship with the server
Runtime `cycles-server` ≥ `0.1.25.10` emits the reservation-lifecycle counters (`cycles_reservations_*_total`, `cycles_events_total`, `cycles_overdraft_incurred_total`). Admin `cycles-server-admin` ≥ `0.1.25.9` emits `cycles_admin_events_emitted_total` and `cycles_admin_webhook_dispatched_total`; `cycles_admin_events_payload_invalid_total` arrived in `0.1.25.12` and `cycles_admin_audit_writes_total` in `0.1.25.20`. All are exposed at `/actuator/prometheus`. See [Custom Cycles metrics](/how-to/observability-setup#custom-cycles-metrics) for the full catalogue (reservation lifecycle, events, overdraft, admin webhooks/events).

The alert rules below use these counters directly where they exist. For signals without a first-class counter (budget utilization, active-reservation count, dispatch-queue depth), derive from balance polling or Redis directly — shown where relevant.
:::

### Prometheus example (using default metrics)

::: warning Percentile histograms are off by default
None of the three services enable Micrometer percentile histograms out of the box, so `http_server_requests_seconds_bucket` series do not exist and `histogram_quantile()` returns nothing. Either enable them (`management.metrics.distribution.percentiles-histogram.http.server.requests=true`) to use the p99 rule below, or alert on `http_server_requests_seconds_max` instead. Also note the `application` tag values: runtime is `cycles-protocol-service`, admin is `cycles-admin-service`, and the events service sets **no** `application` tag — `application=~"cycles-.*"` selectors will not match its series.
:::

```yaml
groups:
  - name: cycles
    rules:
      # Latency — requires percentile histograms enabled (see note above);
      # otherwise use: max_over_time(http_server_requests_seconds_max{...}[5m]) > 0.05
      - alert: CyclesServerLatency
        expr: histogram_quantile(0.99, sum by (le) (rate(http_server_requests_seconds_bucket{application="cycles-protocol-service",uri=~"/v1/reservations.*|/v1/decide"}[5m]))) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Cycles Server p99 latency above 50ms on reservation/decide path"

      # 5xx error rate
      - alert: CyclesServerErrors
        expr: |
          sum(rate(http_server_requests_seconds_count{application=~"cycles-.*",status=~"5.."}[5m]))
            / sum(rate(http_server_requests_seconds_count{application=~"cycles-.*"}[5m]))
            > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Cycles 5xx error rate above 1%"

      # JVM heap pressure
      - alert: CyclesJvmHeapHigh
        expr: |
          jvm_memory_used_bytes{application=~"cycles-.*",area="heap"}
            / jvm_memory_max_bytes{application=~"cycles-.*",area="heap"}
            > 0.8
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "JVM heap usage above 80% for {{ $labels.application }}"
```

### Denial-rate and overdraft alerts (from `cycles_*` counters)

::: tip Why denial rate can't come from `http_server_requests_seconds*`
A live `POST /v1/reservations` denial returns **HTTP 409** with `error: BUDGET_EXCEEDED` — but 409 also covers idempotency mismatches, frozen budgets, and other conflicts, so HTTP status alone can't isolate budget denials. (Only `/v1/decide` and dry-run reserve surface denials as HTTP 200 with `"decision": "DENY"` in the body, which the HTTP histogram can't see at all.) Use the `cycles_reservations_reserve_total` counter instead: its `decision` tag carries `ALLOW`, `ALLOW_WITH_CAPS`, or `DENY`, and `reason` carries the deny code (`BUDGET_EXCEEDED`, `BUDGET_FROZEN`, …). (`ALLOW_WITH_OVERDRAFT` is a value on the separate `overage_policy` tag — the budget's commit-overage policy — not a reservation decision.)
:::

```yaml
- alert: CyclesHighDenialRate
  expr: |
    sum by (tenant) (rate(cycles_reservations_reserve_total{decision="DENY"}[5m]))
    / sum by (tenant) (rate(cycles_reservations_reserve_total[5m]))
    > 0.1
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Over 10% of reservations being denied for {{ $labels.tenant }}"
    # Note: the `reason` label is aggregated away by `sum by (tenant)` and is
    # not available in annotations here. To see top deny reasons, run
    # `topk(5, sum by (reason) (rate(cycles_reservations_reserve_total{decision="DENY"}[5m])))`
    # in the Prometheus UI, or alert per-reason with `sum by (tenant, reason)`.

- alert: CyclesOverdraftSpike
  expr: |
    sum by (tenant) (rate(cycles_overdraft_incurred_total[5m])) > 0
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Tenant {{ $labels.tenant }} incurring overdraft debt for 10m+"

- alert: CyclesReservationExpirySpike
  expr: |
    sum by (tenant) (rate(cycles_reservations_expired_total[5m])) > 1
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Reservation expiry rate elevated for {{ $labels.tenant }} — callers likely failing to commit"
```

### Balance-polling alerts (for signals without a counter)

Some operational questions don't have a direct counter — point-in-time utilization (`spent / allocated`), total debt, and active-reservation counts are all derivable from the ledger but not emitted as gauges. For those, a lightweight sidecar that calls `GET /v1/balances` or `GET /v1/admin/budgets` on a schedule and pushes the sampled values (e.g. `cycles_budget_utilization`, `cycles_budget_debt`) via pushgateway or statsd is the standard pattern. See [Query balances for monitoring](#query-balances-for-monitoring).

### Webhook delivery queue depth

The events service has no `cycles_dispatch_pending_length` gauge yet. Scrape Redis directly with `redis_exporter` — when configured with `--check-single-keys=dispatch:pending`, the exporter exposes the list length as `redis_key_size{key="dispatch:pending"}`:

```yaml
- alert: CyclesWebhookQueueBacklog
  expr: redis_key_size{key="dispatch:pending"} > 100
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Webhook delivery queue depth above 100 — Events Service may be falling behind"
```

For delivery success / failure rates and auto-disabled subscriptions, query the admin API (`GET /v1/admin/webhooks/{id}/deliveries?status=FAILED`) on a schedule and push the sampled counts to your metrics pipeline.

## Dashboard suggestions

::: tip Ready-made operations dashboard
Before building a custom Grafana dashboard, consider the [Cycles Admin Dashboard](/quickstart/deploying-the-cycles-dashboard) — a Vue 3 SPA that ships with an Overview page covering entity counts, top offenders, failing webhooks, and over-limit scopes, plus drill-downs for budgets, events, webhooks, audit, and reservations. It's not a Prometheus dashboard (no time-series charts), but it covers the operator workflows below without any setup. Use it for day-two ops; build the Grafana dashboards described here for time-series alerting and trend analysis.
:::

### Budget overview dashboard

Display for each tenant/scope:

- **Allocated** — total budget
- **Spent** — cumulative spend
- **Reserved** — currently locked by active reservations
- **Remaining** — available for new reservations
- **Debt** — outstanding debt from overdraft commits
- **Utilization %** — gauge showing spent/allocated ratio

### Reservation activity dashboard

- **Reservations/minute** — time series chart showing throughput
- **Decision distribution** — pie chart: ALLOW vs ALLOW_WITH_CAPS vs DENY
- **Avg reservation duration** — time from reserve to commit
- **Expiry rate** — percentage of reservations that expire without commit
- **Top spenders** — table showing which scopes are consuming the most

### Operational health dashboard

- **Server response latency** — p50, p95, p99 time series (Cycles Server + Admin Server)
- **Error rate** — 4xx and 5xx rate across all services
- **Redis connection pool** — active vs available connections
- **Active reservations** — current count (should be bounded)

### Webhook delivery dashboard

- **Queue depth** — `dispatch:pending` length over time (should trend toward zero)
- **Delivery rate** — successful deliveries/minute
- **Retry rate** — retries/minute (indicates endpoint reliability)
- **Failed deliveries** — failed after max retries
- **Auto-disabled subscriptions** — count of subscriptions disabled due to consecutive failures
- **Delivery latency** — time from event to successful delivery (p50, p95)

## Log-based monitoring

If you don't have a metrics pipeline, monitor from server logs:

```bash
# Watch for budget-denied requests (409s are logged as
# "Cycles protocol exception handled: ... error=BUDGET_EXCEEDED ...")
docker compose logs -f cycles-server | grep "error=BUDGET_EXCEEDED"

# Watch for clients hitting already-expired reservations (410s).
# Note: the expiry sweep itself logs successful expirations at DEBUG only —
# use the cycles_reservations_expired_total counter or reservation.expired
# events for expiry-rate monitoring rather than logs.
docker compose logs -f cycles-server | grep "error=RESERVATION_EXPIRED"

# Watch for webhook delivery failures (terminal and transport-level)
docker compose logs -f cycles-events | grep "Webhook delivery permanently failed"
docker compose logs -f cycles-events | grep "Webhook delivery transport failed"

# Watch for auto-disabled subscriptions
docker compose logs -f cycles-events | grep "Webhook subscription auto-disabled"

# Watch for errors across all services
docker compose logs -f cycles-server cycles-admin cycles-events | grep "ERROR"
```

For structured logging, pipe server and SDK application logs to your log aggregation system (ELK, Datadog, CloudWatch). Create alerts for the client recovery signals above as well as server-side errors.

## Next steps

- [Observability Setup](/how-to/observability-setup) — Prometheus, Grafana, and Datadog integration
- [Production Operations Guide](/how-to/production-operations-guide) — deployment and infrastructure
- [Security Hardening](/how-to/security-hardening) — securing the deployment
- [Server Configuration Reference](/configuration/server-configuration-reference-for-cycles) — all server settings



# Multi-Agent Shared Workspace Budget Patterns

When multiple agents operate within the same workspace — a team of planners, executors, and reviewers working on a shared task — they need to share finite reservation capacity without racing each other. This guide covers recommended patterns for structuring budgets in multi-agent systems.

::: warning Concurrency is the core challenge
Multiple agents checking and spending against a shared budget creates race conditions. Always use Cycles reservations (not balance reads) for spending decisions. See [Concurrent Agent Overspend](/incidents/concurrent-agent-overspend) for a detailed explanation of the failure mode.
:::

::: info What “cap” means here
Atomic reservations cap concurrent submitted estimates on mandatory instrumented paths. Actual external cost is known only after execution and settlement follows the configured commit overage policy. Use conservative estimates and treat the application/provider record as the economic outcome.
:::

## Pattern 1: Shared workspace budget with no per-agent limits

The simplest pattern. Every instrumented agent submits the same workspace scope. Atomic reservations prevent their submitted estimates from oversubscribing that explicitly provisioned ledger.

**Scope:** `tenant:acme-corp/workspace:project-alpha`

```bash
# Create a shared $50 budget for the workspace
curl -s -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "scope": "tenant:acme-corp/workspace:project-alpha",
    "unit": "USD_MICROCENTS",
    "allocated": { "amount": 5000000000, "unit": "USD_MICROCENTS" }
  }'
```

**Client setup (Python):**

```python
@cycles(
    estimate=lambda prompt: len(prompt) * 10,
    tenant="acme-corp",
    workspace="project-alpha",
    agent="planner",  # identifies which agent, but all share workspace budget
)
def planner_call(prompt: str) -> str:
    return call_llm(prompt)
```

**When to use:** Small teams of cooperating agents where individual fairness does not matter and one shared estimate-admission ceiling is sufficient.

**Trade-off:** A single expensive agent can exhaust the budget for all others.

## Pattern 2: Per-agent ledgers under a shared workspace cap

Give each agent its own explicitly provisioned ledger, with a workspace ledger acting as a shared cap. These balances are not transferred from the workspace ledger. Each protected call submits both scopes, and the server checks matching ledgers atomically.

**Scope hierarchy:**

```
tenant:acme-corp/workspace:project-alpha          → $50 (shared reservation ceiling)
  tenant:acme-corp/workspace:project-alpha/agent:planner    → $20
  tenant:acme-corp/workspace:project-alpha/agent:executor   → $30
  tenant:acme-corp/workspace:project-alpha/agent:reviewer   → $10
                                                              ───
                                                    Sum: $60 > $50 (OK)
```

```bash
# Workspace-level cap
curl -s -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "scope": "tenant:acme-corp/workspace:project-alpha",
    "unit": "USD_MICROCENTS",
    "allocated": { "amount": 5000000000, "unit": "USD_MICROCENTS" }
  }'

# Per-agent budgets
for agent_budget in "planner:2000000000" "executor:3000000000" "reviewer:1000000000"; do
  agent="${agent_budget%%:*}"
  amount="${agent_budget##*:}"
  curl -s -X POST http://localhost:7979/v1/admin/budgets \
    -H "Content-Type: application/json" \
    -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
    -d "{
      \"scope\": \"tenant:acme-corp/workspace:project-alpha/agent:${agent}\",
      \"unit\": \"USD_MICROCENTS\",
      \"allocated\": { \"amount\": ${amount}, \"unit\": \"USD_MICROCENTS\" }
    }"
done
```

**Why per-agent budgets can exceed the workspace budget:** The workspace scope is checked at reservation time alongside the agent scope. If the workspace is exhausted, the reservation is denied — regardless of the agent's remaining budget. Over-allocating at the agent level provides flexibility: if the planner finishes under budget, the executor can use more of the shared pool.

**When to use:** Multi-agent workflows where you want both individual estimate ceilings and a collective reservation ceiling.

## Pattern 3: Workflow-scoped budgets for task isolation

When agents run multiple independent workflows (e.g. processing different customer requests), scope budgets per workflow to prevent one task from consuming another's budget.

**Scope hierarchy:**

```
tenant:acme-corp/workspace:prod/workflow:task-123       → $10
  tenant:acme-corp/workspace:prod/workflow:task-123/agent:planner
  tenant:acme-corp/workspace:prod/workflow:task-123/agent:executor
tenant:acme-corp/workspace:prod/workflow:task-456       → $10
  tenant:acme-corp/workspace:prod/workflow:task-456/agent:planner
  tenant:acme-corp/workspace:prod/workflow:task-456/agent:executor
```

**Client setup (TypeScript):**

```typescript
const plannerCall = withCycles(
  {
    estimate: 2_000_000,
    tenant: "acme-corp",
    workspace: "prod",
    workflow: taskId,      // dynamic per-task
    agent: "planner",
  },
  async (prompt: string) => callLlm(prompt),
);
```

**When to use:** Task-parallel systems where each task should have an independent budget ceiling.

## Pattern 4: Tiered agents with differentiated limits

Give expensive agents (e.g. those using GPT-4) smaller budgets than cheap agents (e.g. those using GPT-4o-mini), reflecting their different cost profiles.

```bash
# Expensive planner agent — small budget ($5), large model
curl -s -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "scope": "tenant:acme-corp/workspace:prod/agent:planner",
    "unit": "USD_MICROCENTS",
    "allocated": { "amount": 500000000, "unit": "USD_MICROCENTS" }
  }'

# Cheap executor agent — larger budget ($20), smaller model
curl -s -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "scope": "tenant:acme-corp/workspace:prod/agent:executor",
    "unit": "USD_MICROCENTS",
    "allocated": { "amount": 2000000000, "unit": "USD_MICROCENTS" }
  }'
```

## Handling denials gracefully

In any multi-agent system, some agents will be denied budget. Design for this:

```python
from runcycles import cycles, BudgetExceededError

@cycles(estimate=2_000_000, agent="executor")
def executor_call(prompt: str) -> str:
    return call_llm(prompt)

def run_executor(prompt: str) -> str:
    try:
        return executor_call(prompt)
    except BudgetExceededError:
        # Options: use a cheaper model, return cached results,
        # queue for later, or signal the orchestrator to stop
        return use_cheaper_model(prompt)
```

For a full treatment of degradation strategies, see [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer).

## Monitoring shared budgets

Track budget consumption across agents to detect imbalances early:

```bash
# Check remaining budget across all agents in a workspace
curl -s "http://localhost:7878/v1/balances?tenant=acme-corp&workspace=project-alpha" \
  -H "X-Cycles-API-Key: $API_KEY" | jq '.balances[] | {scope_path, remaining: .remaining.amount, spent: .spent.amount}'
```

Set up alerts when any scope drops below 10% remaining with active reservations. See [Monitoring and Alerting](/how-to/monitoring-and-alerting) for detailed setup.

## Key principles

1. **Reserve, don't read.** Balance queries are informational. Reservations are authoritative. Never use a balance read to decide whether to proceed.
2. **Use broader scopes as shared admission ceilings.** Per-agent ledgers provide individual limits; workspace/tenant ledgers constrain combined reservation estimates.
3. **Design for denial.** Any agent can be denied at any time. Graceful degradation is not optional.
4. **Set the `agent` field.** Always identify which agent is spending. This enables per-agent monitoring, debugging, and budget allocation.

## Next steps

- [Common Budget Patterns](/how-to/common-budget-patterns) — per-user, per-workflow, and other scope recipes
- [Concurrent Agent Overspend](/incidents/concurrent-agent-overspend) — the failure mode these patterns prevent
- [Scope Derivation](/protocol/how-scope-derivation-works-in-cycles) — how hierarchical scopes work
- [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) — handling denial gracefully



# Building a Multi-Tenant AI SaaS with Cycles

This guide walks through building a multi-tenant AI SaaS where each customer gets independent budget isolation, plan-tier quotas, and real-time cost visibility. It covers architecture decisions, customer onboarding automation, per-tenant middleware, and operational monitoring.

For individual API details, see [Tenant Management](/how-to/tenant-creation-and-management-in-cycles), [Budget Allocation](/how-to/budget-allocation-and-management-in-cycles), and [Scope Derivation](/protocol/how-scope-derivation-works-in-cycles).

## Architecture: tenant-per-customer

Each customer in your SaaS maps to a Cycles tenant. This gives you:

- **Complete blast-radius isolation** — one customer's runaway agent cannot affect others
- **Independent budget enforcement** — each tenant has its own budget hierarchy
- **Separate API keys** — cryptographic isolation at the protocol level
- **Per-tenant observability** — costs, usage, and denials scoped by customer

```
Your SaaS
├── Customer: Acme Corp        → Tenant: acme
│   ├── Production             → Workspace: prod
│   │   ├── Support bot        → Agent: support-bot
│   │   └── Research agent     → Agent: researcher
│   └── Staging                → Workspace: staging
├── Customer: Globex           → Tenant: globex
│   ├── Production             → Workspace: prod
│   └── Development            → Workspace: dev
```

The full scope hierarchy is: `tenant → workspace → app → workflow → agent → toolset`. Use as many or as few levels as you need — [scope derivation](/protocol/how-scope-derivation-works-in-cycles) handles gap-skipping automatically.

## Plan tiers with budget limits

Map your pricing tiers to budget allocations:

| Plan | Monthly budget | Overdraft | Max agents |
|------|---------------|-----------|------------|
| Free | $5 (500,000,000 microcents) | None | 1 |
| Pro | $50 (5,000,000,000 microcents) | $5 overdraft | 5 |
| Enterprise | $500 (50,000,000,000 microcents) | $50 overdraft | Unlimited |

When a customer hits their budget limit, the next live reservation fails with `409 BUDGET_EXCEEDED` (a `/v1/decide` or dry-run check returns `DENY`). Your application decides what happens: show an upgrade prompt, queue the request, or degrade to a cheaper model. See [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) for patterns.

## Customer onboarding workflow

When a new customer signs up, create their tenant, API key, and initial budget. This is a one-time setup via the Admin API.

### Python

```python
import httpx

ADMIN_URL = "http://localhost:7979"  # Admin Server
ADMIN_KEY = "your-admin-api-key"
CYCLES_KEY = "your-cycles-api-key"   # key with budgets:write permission

# Tenant, API key, and budget PATCH operations use X-Admin-API-Key
admin_headers = {"X-Admin-API-Key": ADMIN_KEY, "Content-Type": "application/json"}
# Budget create/fund operations use X-Cycles-API-Key with budgets:write permission
budget_headers = {"X-Cycles-API-Key": CYCLES_KEY, "Content-Type": "application/json"}

def onboard_customer(customer_id: str, plan: str) -> dict:
    """Create tenant, API key, and budget for a new customer."""

    # 1. Create the tenant (uses admin key)
    tenant_resp = httpx.post(f"{ADMIN_URL}/v1/admin/tenants", headers=admin_headers, json={
        "tenant_id": customer_id,
        "name": f"Customer {customer_id}",
        "metadata": {"plan": plan},
    })
    tenant_resp.raise_for_status()

    # 2. Create an API key for the tenant (uses admin key)
    key_resp = httpx.post(f"{ADMIN_URL}/v1/admin/api-keys", headers=admin_headers, json={
        "tenant_id": customer_id,
        "name": f"{customer_id}-runtime-key",
        "permissions": [
            "reservations:create", "reservations:commit",
            "reservations:release", "reservations:extend",
            "balances:read",
        ],
    })
    key_resp.raise_for_status()
    api_key = key_resp.json()["key_secret"]

    # 3. Create and fund budget ledger (uses cycles key with budgets:write)
    budgets = {
        "free":       {"amount": 500_000_000, "overdraft": 0},
        "pro":        {"amount": 5_000_000_000, "overdraft": 500_000_000},
        "enterprise": {"amount": 50_000_000_000, "overdraft": 5_000_000_000},
    }
    plan_budget = budgets[plan]

    # Create the budget ledger
    httpx.post(f"{ADMIN_URL}/v1/admin/budgets", headers=budget_headers, json={
        "scope": f"tenant:{customer_id}",
        "unit": "USD_MICROCENTS",
        "allocated": {"amount": 0, "unit": "USD_MICROCENTS"},
    }).raise_for_status()

    # Fund it
    scope = f"tenant:{customer_id}"
    httpx.post(
        f"{ADMIN_URL}/v1/admin/budgets/fund?scope={scope}&unit=USD_MICROCENTS",
        headers=budget_headers,
        json={
            "operation": "CREDIT",
            "amount": {"amount": plan_budget["amount"], "unit": "USD_MICROCENTS"},
            "idempotency_key": f"onboard-{customer_id}",
        },
    ).raise_for_status()

    # Set overdraft limit if applicable
    # (PATCH /v1/admin/budgets accepts only X-Admin-API-Key)
    if plan_budget["overdraft"] > 0:
        httpx.patch(
            f"{ADMIN_URL}/v1/admin/budgets?scope={scope}&unit=USD_MICROCENTS",
            headers=admin_headers,
            json={
                "overdraft_limit": {"amount": plan_budget["overdraft"], "unit": "USD_MICROCENTS"},
                "commit_overage_policy": "ALLOW_WITH_OVERDRAFT",
            },
        ).raise_for_status()

    return {"tenant_id": customer_id, "api_key": api_key, "plan": plan}
```

### TypeScript

```typescript
const ADMIN_URL = "http://localhost:7979";
const ADMIN_KEY = "your-admin-api-key";
const CYCLES_KEY = "your-cycles-api-key"; // key with budgets:write permission

// Tenant, API key, and budget PATCH operations use X-Admin-API-Key
const adminHeaders = {
  "X-Admin-API-Key": ADMIN_KEY,
  "Content-Type": "application/json",
};
// Budget create/fund operations use X-Cycles-API-Key with budgets:write permission
const budgetHeaders = {
  "X-Cycles-API-Key": CYCLES_KEY,
  "Content-Type": "application/json",
};

interface OnboardResult {
  tenantId: string;
  apiKey: string;
  plan: string;
}

async function onboardCustomer(
  customerId: string,
  plan: "free" | "pro" | "enterprise",
): Promise<OnboardResult> {
  // 1. Create the tenant (uses admin key)
  const tenantResp = await fetch(`${ADMIN_URL}/v1/admin/tenants`, {
    method: "POST",
    headers: adminHeaders,
    body: JSON.stringify({
      tenant_id: customerId,
      name: `Customer ${customerId}`,
      metadata: { plan },
    }),
  });
  if (!tenantResp.ok) throw new Error(`Tenant creation failed: ${tenantResp.status}`);

  // 2. Create an API key for the tenant (uses admin key)
  const keyResp = await fetch(`${ADMIN_URL}/v1/admin/api-keys`, {
    method: "POST",
    headers: adminHeaders,
    body: JSON.stringify({
      tenant_id: customerId,
      name: `${customerId}-runtime-key`,
      permissions: [
        "reservations:create", "reservations:commit",
        "reservations:release", "reservations:extend",
        "balances:read",
      ],
    }),
  });
  if (!keyResp.ok) throw new Error(`API key creation failed: ${keyResp.status}`);
  const { key_secret: apiKey } = await keyResp.json();

  // 3. Create and fund budget ledger (uses cycles key with budgets:write)
  const budgets = {
    free:       { amount: 500_000_000, overdraft: 0 },
    pro:        { amount: 5_000_000_000, overdraft: 500_000_000 },
    enterprise: { amount: 50_000_000_000, overdraft: 5_000_000_000 },
  };
  const planBudget = budgets[plan];
  const scope = `tenant:${customerId}`;

  // Create the budget ledger
  await fetch(`${ADMIN_URL}/v1/admin/budgets`, {
    method: "POST",
    headers: budgetHeaders,
    body: JSON.stringify({
      scope,
      unit: "USD_MICROCENTS",
      allocated: { amount: 0, unit: "USD_MICROCENTS" },
    }),
  });

  // Fund it
  await fetch(
    `${ADMIN_URL}/v1/admin/budgets/fund?scope=${encodeURIComponent(scope)}&unit=USD_MICROCENTS`,
    {
      method: "POST",
      headers: budgetHeaders,
      body: JSON.stringify({
        operation: "CREDIT",
        amount: { amount: planBudget.amount, unit: "USD_MICROCENTS" },
        idempotency_key: `onboard-${customerId}`,
      }),
    },
  );

  // PATCH /v1/admin/budgets accepts only X-Admin-API-Key
  if (planBudget.overdraft > 0) {
    await fetch(
      `${ADMIN_URL}/v1/admin/budgets?scope=${encodeURIComponent(scope)}&unit=USD_MICROCENTS`,
      {
        method: "PATCH",
        headers: adminHeaders,
        body: JSON.stringify({
          overdraft_limit: { amount: planBudget.overdraft, unit: "USD_MICROCENTS" },
          commit_overage_policy: "ALLOW_WITH_OVERDRAFT",
        }),
      },
    );
  }

  return { tenantId: customerId, apiKey, plan };
}
```

## Per-tenant middleware

Extract the tenant from each request and scope all Cycles operations to that tenant. Store the customer's Cycles API key in your database and use it per-request, or use a shared runtime key with tenant validation.

### FastAPI

```python
from fastapi import Request
from fastapi.responses import JSONResponse

@app.middleware("http")
async def tenant_context(request: Request, call_next):
    tenant = request.headers.get("X-Tenant-ID")
    if not tenant:
        return JSONResponse({"error": "X-Tenant-ID header required"}, status_code=400)
    request.state.tenant = tenant
    return await call_next(request)
```

### Express

```typescript
import { Request, Response, NextFunction } from "express";

function tenantContext(req: Request, res: Response, next: NextFunction) {
  const tenant = req.headers["x-tenant-id"] as string;
  if (!tenant) {
    return res.status(400).json({ error: "X-Tenant-ID header required" });
  }
  res.locals.tenant = tenant;
  next();
}

app.use("/api", tenantContext);
```

### Using tenant in Cycles calls

Pass the tenant to every Cycles operation via the `Subject`:

```python
from fastapi import Request
from runcycles import cycles

@cycles(
    estimate=2_000_000,
    action_kind="llm.completion",
    action_name="gpt-4o",
    # A callable is re-evaluated on every call with the function's
    # arguments — a plain `request.state.tenant` would be captured
    # once at decoration time and pin every call to the same tenant.
    tenant=lambda request, prompt: request.state.tenant,
)
async def handle_chat(request: Request, prompt: str) -> dict:
    ...
```

Or with the programmatic client:

```python
from runcycles import Subject

subject = Subject(
    tenant=request.state.tenant,
    workspace="prod",
    agent="support-bot",
)
```

## Per-workspace environment isolation

Split each tenant's budget across environments to prevent staging and development from consuming production budget:

```python
def setup_workspace_budgets(customer_id: str, total_budget: int):
    """Split tenant budget: 80% prod, 15% staging, 5% dev."""
    allocations = {
        "prod": int(total_budget * 0.80),
        "staging": int(total_budget * 0.15),
        "dev": int(total_budget * 0.05),
    }

    for workspace, amount in allocations.items():
        scope = f"tenant:{customer_id}/workspace:{workspace}"

        # Create the workspace budget ledger
        httpx.post(f"{ADMIN_URL}/v1/admin/budgets", headers=budget_headers, json={
            "scope": scope,
            "unit": "USD_MICROCENTS",
            "allocated": {"amount": 0, "unit": "USD_MICROCENTS"},
        }).raise_for_status()

        # Fund it
        httpx.post(
            f"{ADMIN_URL}/v1/admin/budgets/fund?scope={scope}&unit=USD_MICROCENTS",
            headers=budget_headers,
            json={
                "operation": "CREDIT",
                "amount": {"amount": amount, "unit": "USD_MICROCENTS"},
                "idempotency_key": f"ws-{customer_id}-{workspace}",
            },
        ).raise_for_status()
```

Now development runaway loops burn the dev budget ($0.25), not the production budget ($4.00).

## Plan upgrades and downgrades

When a customer changes plans, adjust their budget:

```python
def upgrade_plan(customer_id: str, old_plan: str, new_plan: str):
    """Credit the difference between plans."""
    budgets = {
        "free": 500_000_000,
        "pro": 5_000_000_000,
        "enterprise": 50_000_000_000,
    }
    difference = budgets[new_plan] - budgets[old_plan]
    scope = f"tenant:{customer_id}"

    if difference > 0:
        # Upgrade: credit the difference
        httpx.post(
            f"{ADMIN_URL}/v1/admin/budgets/fund?scope={scope}&unit=USD_MICROCENTS",
            headers=budget_headers,
            json={
                "operation": "CREDIT",
                "amount": {"amount": difference, "unit": "USD_MICROCENTS"},
                "idempotency_key": f"upgrade-{customer_id}-{new_plan}",
            },
        ).raise_for_status()

    # Update tenant metadata
    httpx.patch(
        f"{ADMIN_URL}/v1/admin/tenants/{customer_id}",
        headers=admin_headers,
        json={"metadata": {"plan": new_plan}},
    ).raise_for_status()
```

For downgrades, the remaining budget stays as-is until the billing period resets. Use `RESET_SPENT` at the start of each billing cycle to clear the previous period's consumption and set the new plan's allocation in one step.

## Monthly budget reset

At the start of each billing period, reset budgets to the plan allocation with `RESET_SPENT`:

```python
from datetime import datetime, timezone

def monthly_reset(customer_id: str, plan: str):
    """Reset tenant budget for the new billing cycle."""
    budgets = {"free": 500_000_000, "pro": 5_000_000_000, "enterprise": 50_000_000_000}
    scope = f"tenant:{customer_id}"
    period = datetime.now(timezone.utc).strftime("%Y-%m")  # e.g. "2026-07"

    httpx.post(
        f"{ADMIN_URL}/v1/admin/budgets/fund?scope={scope}&unit=USD_MICROCENTS",
        headers=budget_headers,
        json={
            "operation": "RESET_SPENT",
            "amount": {"amount": budgets[plan], "unit": "USD_MICROCENTS"},
            "idempotency_key": f"reset-{customer_id}-{plan}-{period}",
            "reason": "Monthly billing period reset",
        },
    ).raise_for_status()
```

Run this from a cron job or billing system webhook. `RESET_SPENT` sets allocated to the new amount and clears spent to 0 (debt and active reservations carry forward); emits a `budget.reset_spent` event that your dashboards can route as a period-boundary signal distinct from ceiling adjustments.

## Customer-facing usage dashboard

Expose per-tenant budget data so customers can see their own usage:

```python
from fastapi import Request

@app.get("/api/usage")
async def usage(request: Request):
    tenant = request.state.tenant
    client = request.app.state.cycles_client

    response = client.get_balances(tenant=tenant)
    if not response.is_success:
        return JSONResponse({"error": "Failed to fetch usage"}, status_code=500)

    balances = response.body.get("balances", [])
    return {
        "tenant": tenant,
        "balances": [
            {
                "scope": b.get("scope"),
                "allocated": b.get("allocated", {}).get("amount"),
                "spent": b.get("spent", {}).get("amount"),
                "remaining": b.get("remaining", {}).get("amount"),
                "unit": b.get("allocated", {}).get("unit"),
                "is_over_limit": b.get("is_over_limit", False),
            }
            for b in balances
        ],
    }
```

## Handling budget exhaustion gracefully

When a customer hits their limit, your application should degrade gracefully rather than crash:

```python
from runcycles import BudgetExceededError

async def handle_request(prompt: str, tenant: str) -> dict:
    try:
        return await guarded_llm_call(prompt, tenant=tenant)
    except BudgetExceededError:
        # Option 1: Return a friendly error
        return {
            "content": None,
            "error": "budget_exceeded",
            "message": "You've used your monthly AI budget. Upgrade your plan or wait for the next billing cycle.",
            "upgrade_url": f"/billing/upgrade?tenant={tenant}",
        }
```

For more strategies, see [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer):

| Strategy | When to use |
|----------|-------------|
| **Show upgrade prompt** | Free-tier users hitting limits |
| **Queue for later** | Batch workloads that can wait |
| **Downgrade model** | Use a cheaper model (GPT-4o-mini instead of GPT-4o) |
| **Cache responses** | Repeat queries that have been answered before |
| **Disable feature** | Turn off expensive features while keeping basic ones |

## Suspending and closing tenants

When a customer churns or violates terms, suspend or close their tenant:

```python
def suspend_customer(customer_id: str):
    """Suspend: blocks new reservations, allows existing to complete."""
    httpx.patch(f"{ADMIN_URL}/v1/admin/tenants/{customer_id}", headers=admin_headers, json={
        "status": "SUSPENDED",
    }).raise_for_status()

def close_customer(customer_id: str):
    """Close: blocks all operations. Irreversible."""
    httpx.patch(f"{ADMIN_URL}/v1/admin/tenants/{customer_id}", headers=admin_headers, json={
        "status": "CLOSED",
    }).raise_for_status()
```

**SUSPENDED** blocks new reservations but lets in-flight work complete. **CLOSED** blocks everything and is irreversible. Use `SUSPENDED` first to allow graceful wind-down.

## Monitoring per-tenant health

Set up alerts for per-tenant budget exhaustion using [Webhook Integrations](/how-to/webhook-integrations):

```python
# Example: Slack alert when a tenant crosses 80% budget utilization
# Configure via Admin API webhook:
# POST /v1/admin/webhooks
# {
#   "url": "https://hooks.slack.com/services/...",
#   "event_types": ["budget.threshold_crossed"],
#   "thresholds": {"budget_utilization": [0.80]}
# }
```

Key metrics to monitor per tenant:

| Metric | Alert threshold | Action |
|--------|----------------|--------|
| Budget utilization | > 80% | Notify customer, suggest upgrade |
| Budget utilization | > 95% | Internal alert, prepare for denial |
| Denial rate | > 10% | Customer likely hitting limits — outreach |
| Tenant status | SUSPENDED | Investigate, notify billing team |

See [Monitoring and Alerting](/how-to/monitoring-and-alerting) for PromQL queries and Grafana dashboards.

## Troubleshooting

### Common multi-tenant mistakes

**Inconsistent tenant IDs.** If some requests pass `tenant: "acme"` and others pass `tenant: "Acme"`, they hit different budget scopes. Normalize tenant IDs to lowercase at the middleware level.

**Missing X-Tenant-ID header.** Without tenant extraction, all requests share the default tenant's budget. Use middleware that rejects requests without the header.

**Shared API key across tenants.** Each API key is bound to one tenant. If you use a shared key, all requests are attributed to that key's tenant. Use per-tenant API keys for proper isolation.

**Budget allocated at wrong scope.** If you allocate budget at `tenant:acme` but your agents report with `tenant:acme/workspace:prod`, the workspace-level budget is missing and enforcement is skipped at that level. Allocate at the levels you want to enforce.

## Key points

- **One tenant per customer.** Map each SaaS customer to a Cycles tenant for complete budget isolation.
- **Plan tiers map to budget allocations.** Free, Pro, Enterprise plans differ in allocated budget and overdraft limits.
- **Automate onboarding.** Create tenant, API key, and budget in a single workflow when a customer signs up.
- **Extract tenant from every request.** Use middleware to ensure every Cycles call is scoped to the requesting customer.
- **Reset monthly.** Use `RESET_SPENT` at billing-cycle boundaries (clears spent for the new period). Reserve `RESET` for plan-ceiling changes — it preserves spent.
- **Degrade gracefully.** When budget is exhausted, show upgrade prompts or switch to cheaper models.
- **Monitor per-tenant.** Alert on budget thresholds before customers are affected.

## Next steps

- [Tenant Creation and Management](/how-to/tenant-creation-and-management-in-cycles) — Admin API for tenant lifecycle
- [Budget Allocation and Management](/how-to/budget-allocation-and-management-in-cycles) — credit, debit, reset operations
- [Scope Derivation](/protocol/how-scope-derivation-works-in-cycles) — how budget hierarchies work
- [API Key Management](/how-to/api-key-management-in-cycles) — per-tenant key creation and permissions
- [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) — strategies for budget exhaustion
- [Webhook Integrations](/how-to/webhook-integrations) — per-tenant alerting via Slack, PagerDuty
- [Common Budget Patterns](/how-to/common-budget-patterns) — reusable budget recipes
- [Monitoring and Alerting](/how-to/monitoring-and-alerting) — operational dashboards



# Observability Setup

This guide covers how to expose metrics from the Cycles Server and visualize them in Prometheus, Grafana, and Datadog. For alerting rules and budget-level monitoring patterns, see [Monitoring and Alerting](/how-to/monitoring-and-alerting).

## Exposing Prometheus metrics

The Cycles Server is a Spring Boot application. To expose Prometheus-format metrics, enable the Actuator Prometheus endpoint.

### Step 1: Enable the Prometheus endpoint

Set the following property via environment variable or `application.properties`:

```properties
management.endpoints.web.exposure.include=health,info,prometheus
```

In Docker Compose:

```yaml
cycles-server:
  image: ghcr.io/runcycles/cycles-server:0.1.25.59
  environment:
    REDIS_HOST: redis
    REDIS_PORT: 6379
    REDIS_PASSWORD: ${REDIS_PASSWORD}
    MANAGEMENT_ENDPOINTS_WEB_EXPOSURE_INCLUDE: health,info,prometheus
  ports:
    - "7878:7878"
```

### Step 2: Verify

Since `cycles-server` 0.1.25.45, `/actuator/prometheus` (and the aggregate `/actuator/health`) require the `X-Admin-API-Key` header on the runtime and admin servers; only `/actuator/health/liveness` and `/actuator/health/readiness` remain public. The events service's management port (9980) has no auth filter in the reference deployment — restrict it at the network layer.

```bash
curl -s -H "X-Admin-API-Key: $ADMIN_KEY" http://localhost:7878/actuator/prometheus | head -20
```

You should see Micrometer metrics in Prometheus exposition format:

```
# HELP http_server_requests_seconds Duration of HTTP server request handling
# TYPE http_server_requests_seconds summary
http_server_requests_seconds_count{method="POST",uri="/v1/reservations",status="200"} 142.0
...
```

(A successful reserve returns HTTP **200**. `_bucket` histogram series only appear if you enable percentile histograms — see the latency note below.)

### Step 3: Configure Prometheus scrape

Add the Cycles Server as a target in your `prometheus.yml`. Because the runtime and admin scrape endpoints require `X-Admin-API-Key` (0.1.25.45+), the scrape config must send that header — Prometheus supports custom headers via `http_headers` (Prometheus v3.0+; on older versions, front the endpoint with a proxy that injects the header):

```yaml
scrape_configs:
  - job_name: "cycles-server"
    metrics_path: "/actuator/prometheus"
    scrape_interval: 15s
    http_headers:
      X-Admin-API-Key:
        secrets: ["${ADMIN_API_KEY}"]
    static_configs:
      - targets: ["cycles-server:7878"]
        labels:
          service: "cycles"

  # Optional: scrape the Admin Server too
  - job_name: "cycles-admin"
    metrics_path: "/actuator/prometheus"
    scrape_interval: 30s
    http_headers:
      X-Admin-API-Key:
        secrets: ["${ADMIN_API_KEY}"]
    static_configs:
      - targets: ["cycles-admin:7979"]
        labels:
          service: "cycles-admin"

  # Events service — management port 9980, no admin-key requirement
  - job_name: "cycles-events"
    metrics_path: "/actuator/prometheus"
    scrape_interval: 30s
    static_configs:
      - targets: ["cycles-events:9980"]
        labels:
          service: "cycles-events"
```

## Key metrics reference

The Cycles Server exposes standard Spring Boot Actuator / Micrometer metrics. These are the most relevant for Cycles:

### HTTP endpoint metrics

| Metric | Type | Description |
|---|---|---|
| `http_server_requests_seconds` | histogram | Request duration by `method`, `uri`, `status` |
| `http_server_requests_seconds_count` | counter | Total request count by `method`, `uri`, `status` |

Key `uri` labels for Cycles endpoints:

| URI pattern | Operation |
|---|---|
| `/v1/reservations` | Create reservation |
| `/v1/reservations/{id}/commit` | Commit |
| `/v1/reservations/{id}/release` | Release |
| `/v1/reservations/{id}/extend` | Heartbeat extend |
| `/v1/decide` | Preflight decision |
| `/v1/events` | Direct debit event |
| `/v1/balances` | Balance query |

### JVM metrics

| Metric | Description |
|---|---|
| `jvm_memory_used_bytes{area="heap"}` | Current heap usage |
| `jvm_memory_max_bytes{area="heap"}` | Maximum heap size |
| `jvm_gc_pause_seconds` | GC pause duration |
| `jvm_threads_live_threads` | Active thread count |

### System metrics

| Metric | Description |
|---|---|
| `system_cpu_usage` | System CPU utilization (0.0–1.0) |
| `process_cpu_usage` | Process CPU utilization (0.0–1.0) |

### Custom Cycles metrics

The runtime server (`cycles-server` ≥ `0.1.25.10`) emits custom Micrometer counters under the `cycles.*` namespace, exposed in Prometheus format as `cycles_*`:

| Metric | Tags | Description |
|---|---|---|
| `cycles_reservations_reserve_total` | `tenant`, `decision`, `reason`, `overage_policy` | Outcome of every `POST /v1/reservations` call. `decision=ALLOW\|ALLOW_WITH_CAPS\|DENY`; `reason` carries the deny/caps code; `overage_policy` carries the budget's commit-overage policy (`REJECT`, `ALLOW_IF_AVAILABLE`, `ALLOW_WITH_OVERDRAFT`). |
| `cycles_reservations_commit_total` | `tenant`, `decision`, `reason`, `overage_policy` | Outcome of every commit. `decision=COMMITTED\|DENY`. |
| `cycles_reservations_release_total` | `tenant`, `actor_type`, `decision`, `reason` | Every successful release. `actor_type` distinguishes tenant-driven from admin-on-behalf-of releases. |
| `cycles_reservations_extend_total` | `tenant`, `decision`, `reason` | Every extend attempt. |
| `cycles_reservations_expired_total` | `tenant` | Per reservation actually marked EXPIRED by the sweep (not per candidate). |
| `cycles_reservations_quarantined_total` | `tenant`, `reason` | Malformed reservation records quarantined by maintenance. |
| `cycles_reservations_created_at_index_reads_total` | `outcome` | Created-at index reads and completeness-gated fallback outcomes. |
| `cycles_maintenance_runs_total` | `job`, `outcome` | Scheduled maintenance run outcomes. |
| `cycles_maintenance_duration_seconds` | `job`, `outcome` | Scheduled maintenance duration timer. |
| `cycles_events_total` | `tenant`, `decision`, `reason`, `overage_policy` | Outcome of every `POST /v1/events` one-shot debit. |
| `cycles_overdraft_incurred_total` | `tenant` | Count of commits/events that actually accrued non-zero debt (unit-free — amount is in the balance store, not leaked to metrics). |
| `cycles_evidence_emit_failed_total` | `artifact_type` | Evidence-source enqueue failures (fail-open) — the rare loss window where a lifecycle op committed but its evidence record could not be queued. |

The admin server currently exposes seven custom counters:

| Metric | Description |
|---|---|
| `cycles_admin_webhook_dispatched_total` | Webhook-delivery enqueue outcomes (`result=queued`/`failure`/`boundary_skipped`). |
| `cycles_admin_events_emitted_total` | Events produced by admin controllers (budget/tenant/policy/api_key/system). |
| `cycles_admin_events_payload_invalid_total` | Payload contract violations caught at emit time. |
| `cycles_admin_audit_writes_total` | Audit-write attempts by `path_class` and `outcome` (`written`/`error`/`sampled-out`). Alert on any `outcome="error"`. |
| `cycles_admin_tenant_close_outbox_dead_letter_total` | Tenant-close outbox dead letters by `resource_type`. |
| `cycles_admin_tenant_close_reconcile_incomplete_total` | Reconciliation runs that ended with incomplete tenant-close work. |
| `cycles_admin_tenant_close_reconcile_errors_total` | Tenant-close reconciliation errors. |

The events service adds 17 delivery, evidence, dispatcher, and security counters plus one delivery-latency timer. Use the [Prometheus Metrics Reference](/how-to/prometheus-metrics-reference) for the complete names, labels, and enum values rather than copying a partial inventory into dashboards.

The high-cardinality `tenant` tag is controlled per service by `cycles.metrics.tenant-tag.enabled`: the runtime server defaults it to **`true`**, the events service defaults it to **`false`**, and the admin `cycles_admin_*` counters carry no tenant tag at all. Disable it on the runtime server in deployments with many thousands of tenants. Empty/null tag values are normalised to the sentinel `UNKNOWN` so series names stay stable.

For denial-rate, overdraft-rate, and tenant-level alerts, prefer these `cycles_*` counters over `http_server_requests_seconds_count` — a live reserve denial is an HTTP 409, but 409 also covers idempotency mismatches and frozen budgets, and `/v1/decide`/dry-run denials are HTTP 200 with `decision: DENY` in the body, which HTTP metrics can't see. Status codes alone would miscount.

## PromQL query cookbook

### Reservation throughput (requests/second)

```promql
rate(http_server_requests_seconds_count{uri="/v1/reservations",method="POST"}[5m])
```

### Commit throughput

```promql
rate(http_server_requests_seconds_count{uri=~"/v1/reservations/.+/commit",method="POST"}[5m])
```

### Reservation latency (p50, p95, p99)

::: warning Requires percentile histograms
The quantile queries below (and the latency panels in the Grafana dashboard) rely on `http_server_requests_seconds_bucket` series, which none of the Cycles services publish by default. Enable them with `management.metrics.distribution.percentiles-histogram.http.server.requests=true` (env var `MANAGEMENT_METRICS_DISTRIBUTION_PERCENTILES_HISTOGRAM_HTTP_SERVER_REQUESTS=true`). Without that, use `http_server_requests_seconds_max` and the `_sum`/`_count` average.
:::

```promql
# p50
histogram_quantile(0.5, rate(http_server_requests_seconds_bucket{uri="/v1/reservations",method="POST"}[5m]))

# p95
histogram_quantile(0.95, rate(http_server_requests_seconds_bucket{uri="/v1/reservations",method="POST"}[5m]))

# p99
histogram_quantile(0.99, rate(http_server_requests_seconds_bucket{uri="/v1/reservations",method="POST"}[5m]))
```

### Denial rate (409 responses on reservation create)

```promql
rate(http_server_requests_seconds_count{uri="/v1/reservations",method="POST",status="409"}[5m])
/
rate(http_server_requests_seconds_count{uri="/v1/reservations",method="POST"}[5m])
```

This is an approximation: 409 also covers idempotency mismatches and frozen budgets, and it misses `/v1/decide`/dry-run denials (HTTP 200 with `decision: DENY`). For an exact denial rate, use `cycles_reservations_reserve_total{decision="DENY"}` — see [Custom Cycles metrics](#custom-cycles-metrics).

### Server error rate (5xx)

```promql
sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
/
sum(rate(http_server_requests_seconds_count[5m]))
```

### JVM heap utilization

```promql
jvm_memory_used_bytes{area="heap"} / jvm_memory_max_bytes{area="heap"}
```

## Grafana dashboard

Import the following JSON into Grafana (**Dashboards > Import > Paste JSON**). It creates a "Cycles Overview" dashboard with three rows: throughput, latency, and infrastructure. The two latency panels use `histogram_quantile` over `_bucket` series, so they require percentile histograms to be enabled (see the warning above); the other panels work with the default metric set.

::: details Click to expand dashboard JSON
```json
{
  "dashboard": {
    "title": "Cycles Overview",
    "tags": ["cycles"],
    "timezone": "browser",
    "refresh": "30s",
    "panels": [
      {
        "title": "Reservation Throughput",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 8, "x": 0, "y": 0 },
        "targets": [{
          "expr": "rate(http_server_requests_seconds_count{uri=\"/v1/reservations\",method=\"POST\"}[5m])",
          "legendFormat": "reservations/sec"
        }]
      },
      {
        "title": "Commit Throughput",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 8, "x": 8, "y": 0 },
        "targets": [{
          "expr": "rate(http_server_requests_seconds_count{uri=~\"/v1/reservations/.+/commit\",method=\"POST\"}[5m])",
          "legendFormat": "commits/sec"
        }]
      },
      {
        "title": "Denial Rate",
        "type": "gauge",
        "gridPos": { "h": 8, "w": 8, "x": 16, "y": 0 },
        "targets": [{
          "expr": "rate(http_server_requests_seconds_count{uri=\"/v1/reservations\",method=\"POST\",status=\"409\"}[5m]) / rate(http_server_requests_seconds_count{uri=\"/v1/reservations\",method=\"POST\"}[5m])",
          "legendFormat": "denial rate"
        }],
        "fieldConfig": {
          "defaults": {
            "unit": "percentunit",
            "thresholds": {
              "steps": [
                { "color": "green", "value": 0 },
                { "color": "yellow", "value": 0.05 },
                { "color": "red", "value": 0.1 }
              ]
            }
          }
        }
      },
      {
        "title": "Reservation Latency",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
        "targets": [
          {
            "expr": "histogram_quantile(0.5, rate(http_server_requests_seconds_bucket{uri=\"/v1/reservations\",method=\"POST\"}[5m]))",
            "legendFormat": "p50"
          },
          {
            "expr": "histogram_quantile(0.95, rate(http_server_requests_seconds_bucket{uri=\"/v1/reservations\",method=\"POST\"}[5m]))",
            "legendFormat": "p95"
          },
          {
            "expr": "histogram_quantile(0.99, rate(http_server_requests_seconds_bucket{uri=\"/v1/reservations\",method=\"POST\"}[5m]))",
            "legendFormat": "p99"
          }
        ],
        "fieldConfig": { "defaults": { "unit": "s" } }
      },
      {
        "title": "Commit Latency",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
        "targets": [
          {
            "expr": "histogram_quantile(0.5, rate(http_server_requests_seconds_bucket{uri=~\"/v1/reservations/.+/commit\",method=\"POST\"}[5m]))",
            "legendFormat": "p50"
          },
          {
            "expr": "histogram_quantile(0.95, rate(http_server_requests_seconds_bucket{uri=~\"/v1/reservations/.+/commit\",method=\"POST\"}[5m]))",
            "legendFormat": "p95"
          },
          {
            "expr": "histogram_quantile(0.99, rate(http_server_requests_seconds_bucket{uri=~\"/v1/reservations/.+/commit\",method=\"POST\"}[5m]))",
            "legendFormat": "p99"
          }
        ],
        "fieldConfig": { "defaults": { "unit": "s" } }
      },
      {
        "title": "Error Rate (5xx)",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 8, "x": 0, "y": 16 },
        "targets": [{
          "expr": "sum(rate(http_server_requests_seconds_count{status=~\"5..\"}[5m])) / sum(rate(http_server_requests_seconds_count[5m]))",
          "legendFormat": "5xx rate"
        }],
        "fieldConfig": { "defaults": { "unit": "percentunit" } }
      },
      {
        "title": "JVM Heap Usage",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 8, "x": 8, "y": 16 },
        "targets": [
          {
            "expr": "jvm_memory_used_bytes{area=\"heap\"}",
            "legendFormat": "used"
          },
          {
            "expr": "jvm_memory_max_bytes{area=\"heap\"}",
            "legendFormat": "max"
          }
        ],
        "fieldConfig": { "defaults": { "unit": "bytes" } }
      },
      {
        "title": "CPU Usage",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 8, "x": 16, "y": 16 },
        "targets": [
          {
            "expr": "process_cpu_usage",
            "legendFormat": "process"
          },
          {
            "expr": "system_cpu_usage",
            "legendFormat": "system"
          }
        ],
        "fieldConfig": { "defaults": { "unit": "percentunit" } }
      }
    ],
    "schemaVersion": 39
  },
  "overwrite": true
}
```
:::

After importing, set the **Prometheus** data source if prompted.

## Datadog integration

### Option A: Datadog Agent with Spring Boot integration

If you run the Datadog Agent alongside the Cycles Server, enable the Spring Boot Actuator check:

```yaml
# datadog-agent/conf.d/openmetrics.d/conf.yaml
instances:
  - openmetrics_endpoint: http://cycles-server:7878/actuator/prometheus
    namespace: cycles
    metrics:
      - http_server_requests_seconds
      - jvm_memory_used_bytes
      - jvm_gc_pause_seconds
      - system_cpu_usage
      - process_cpu_usage
```

### Option B: Micrometer Datadog registry

Add the `micrometer-registry-datadog` dependency to the Cycles Server and configure:

```properties
management.datadog.metrics.export.api-key=${DD_API_KEY}
management.datadog.metrics.export.step=30s
management.datadog.metrics.export.uri=https://api.datadoghq.com
```

### Key Datadog monitors

| Monitor | Query | Threshold |
|---|---|---|
| Reservation latency | `avg:cycles.http_server_requests_seconds.p99{uri:/v1/reservations}` | > 0.05 (50ms) |
| Error rate | `sum:cycles.http_server_requests_seconds_count{status:5*}.as_rate() / sum:cycles.http_server_requests_seconds_count{*}.as_rate()` | > 0.01 (1%) |
| JVM heap | `avg:cycles.jvm_memory_used_bytes{area:heap} / avg:cycles.jvm_memory_max_bytes{area:heap}` | > 0.8 (80%) |

## Client-side observability

### Logging

All four clients expose lifecycle and recovery diagnostics through their language's logging surface:

- **Python**: Set `logging.getLogger("runcycles").setLevel(logging.DEBUG)`
- **TypeScript**: The client reports heartbeat and retained-settlement warnings via `console.warn`
- **Spring Boot**: Set `logging.level.io.runcycles=DEBUG` in `application.yml`
- **Rust**: Install a `tracing` subscriber and enable the `runcycles` target

Collect warnings for journal I/O failures, quarantined records, retry exhaustion, authentication failures, expired-commit event fallback, and heartbeat stop dispositions. See [Monitoring and Alerting](/how-to/monitoring-and-alerting#client-recovery-signals).

### Custom instrumentation with OpenTelemetry

Wrap the decorator or HOF with your own spans to trace reservation lifecycles in your distributed tracing system:

```python
from opentelemetry import trace
from runcycles import cycles

tracer = trace.get_tracer("my-app")

@cycles(estimate=1000, action_kind="llm.completion", action_name="gpt-4o")
def call_llm(prompt: str) -> str:
    with tracer.start_as_current_span("cycles.call_llm") as span:
        span.set_attribute("cycles.estimate", 1000)
        result = openai.chat.completions.create(model="gpt-4o", messages=[{"role": "user", "content": prompt}])
        span.set_attribute("cycles.actual_tokens", result.usage.total_tokens)
        return result.choices[0].message.content
```

For structured error logging patterns, see [Error Handling Patterns](/how-to/error-handling-patterns-in-cycles-client-code).

## Next steps

- [Monitoring and Alerting](/how-to/monitoring-and-alerting) — alerting rules and budget monitoring patterns
- [Client Performance Tuning](/how-to/client-performance-tuning) — timeout and retry optimization
- [Production Operations Guide](/how-to/production-operations-guide) — server infrastructure
- [Server Configuration Reference](/configuration/server-configuration-reference-for-cycles) — all server properties



# Production Operations Guide

This guide covers what you need to run Cycles reliably in production. It assumes you've already deployed the stack per [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack) and are preparing for production traffic.

::: info
Cycles stores all state in Redis. Redis availability directly determines Cycles availability. Plan your Redis deployment accordingly.
:::

::: tip Operations UI for incident response
For incident-response workflows — freeze a runaway budget, suspend a tenant, force-release hung reservations, replay missed webhooks, revoke a leaked API key — deploy the [Cycles Admin Dashboard](/quickstart/deploying-the-cycles-dashboard). It's a Vue 3 SPA with one-click actions (capability-gated, with confirm + blast-radius summaries) that's typically faster than crafting curl during a live incident. Pair with the Prometheus alerting in [Monitoring and Alerting](/how-to/monitoring-and-alerting) — alerts page you, dashboard helps you act.
:::

## Redis configuration for production

Cycles stores all state in Redis. Redis availability directly determines Cycles availability.

::: warning Always configure Redis authentication in production
Set `REDIS_PASSWORD` and provide it to all Cycles services. An unauthenticated Redis instance is a critical security vulnerability — anyone with network access can read budget state, modify reservations, and extract API keys. See [Security Hardening — Redis Authentication](/how-to/security-hardening#authentication) for complete setup including TLS and ACLs.
:::

### Persistence

Enable both RDB snapshots and AOF append-only logging:

```conf
# redis.conf
save 900 1        # Snapshot every 15 min if at least 1 key changed
save 300 10       # Snapshot every 5 min if at least 10 keys changed
appendonly yes     # Enable AOF
appendfsync everysec  # Fsync once per second (good balance of safety and performance)
```

In Docker Compose:

```yaml
redis:
  image: redis:7-alpine
  command: redis-server --appendonly yes --save "900 1" --save "300 10"
  volumes:
    - redis-data:/data
```

### Memory management

Set a max memory limit and eviction policy:

```conf
maxmemory 2gb
maxmemory-policy noeviction  # IMPORTANT: never evict budget data
```

**Always use `noeviction`**. Evicting budget keys silently loses budget state. It is better for Redis to reject writes (causing reservation failures that can be retried) than to silently drop data.

### High availability

For production, consider:

- **Redis Sentinel** — automatic failover with a primary + replica setup. Good for most deployments.
- **Redis Cluster** — sharded across multiple nodes. Required for very large deployments.

Cycles uses Lua scripts for atomic operations. All keys for a single reservation operation are in the same Redis keyspace, so single-instance and Sentinel setups work out of the box. For Redis Cluster, ensure the key prefix strategy keeps related keys on the same shard.

### Backup strategy

- **Automated RDB snapshots** stored offsite (S3, GCS, etc.)
- **AOF backups** for point-in-time recovery
- **Test restores regularly** — untested backups are not backups

Use the [Redis Backup, Restore, and Disaster Recovery Runbook](/how-to/redis-backup-restore-disaster-recovery) for a Cycles-aware procedure. Restoring Redis rolls back budget balances, reservations, idempotency records, webhook queues, audit records, and evidence state together; do not restore selected key families independently.

## Cycles Server configuration

### Running multiple instances

The Cycles Server is stateless. You can run multiple instances behind a load balancer:

```yaml
cycles-server-1:
  image: ghcr.io/runcycles/cycles-server:0.1.25.59
  environment:
    REDIS_HOST: redis-primary
    REDIS_PORT: 6379
    REDIS_PASSWORD: ${REDIS_PASSWORD}

cycles-server-2:
  image: ghcr.io/runcycles/cycles-server:0.1.25.59
  environment:
    REDIS_HOST: redis-primary
    REDIS_PORT: 6379
    REDIS_PASSWORD: ${REDIS_PASSWORD}
```

Any load balancing strategy works (round-robin, least-connections). No sticky sessions required.

### Health checks

All three services (runtime, admin, events) enable Spring's dedicated Kubernetes liveness/readiness probes (`management.endpoint.health.probes.enabled=true`) and serve them at `/actuator/health/liveness` and `/actuator/health/readiness`. The probe endpoints are public (unauthenticated). Since 0.1.25.45, all **other** actuator endpoints — the aggregate `/actuator/health`, `/actuator/info`, and `/actuator/prometheus` — require the admin API key via the `X-Admin-API-Key` header.

```bash
# Cycles Server (Kubernetes probes, public)
curl http://localhost:7878/actuator/health/liveness
curl http://localhost:7878/actuator/health/readiness

# Admin Server (Kubernetes probes, public)
curl http://localhost:7979/actuator/health/liveness
curl http://localhost:7979/actuator/health/readiness

# Events Service (Kubernetes probes, management port)
curl http://localhost:9980/actuator/health/liveness
curl http://localhost:9980/actuator/health/readiness

# Aggregate health requires the admin API key (since 0.1.25.45)
curl -H "X-Admin-API-Key: $ADMIN_KEY" http://localhost:7878/actuator/health
```

Configure your load balancer or orchestrator to check these endpoints. On Kubernetes, wire liveness probes to `/actuator/health/liveness` and readiness probes to `/actuator/health/readiness` on all three services — the runtime service on port 7878, the Admin Server on port 7979, and the Events Service on its management port 9980, not the app port 7980. Readiness includes a Redis `PING` health contributor and turns `DOWN` when Redis is unreachable; liveness stays process-only. There is no custom queue-consumption health check on the Events Service today; for backlog monitoring, watch `LLEN dispatch:pending` (see [Monitoring and Alerting](/how-to/monitoring-and-alerting)).

### JVM tuning

The default JVM settings work for most deployments. For high-throughput environments:

```bash
JAVA_OPTS="-Xms512m -Xmx1g -XX:+UseG1GC"
```

### Reservation expiry

The server runs a background sweep to expire stale reservations:

```yaml
cycles:
  expiry:
    interval-ms: 5000  # Default: sweep every 5 seconds
```

Reduce the interval for tighter TTL enforcement. Increase it to reduce Redis load if TTL precision is not critical.

For listing and recovering stale or orphaned reservations after client crashes, see [Reservation Recovery and Listing](/protocol/reservation-recovery-and-listing-in-cycles).

## Events Service configuration

The **Cycles Events Service** (`cycles-server-events`) delivers webhook notifications asynchronously and signs CyclesEvidence envelopes when evidence is enabled. It is an outbound worker: its app port 7980 and management port 9980 should stay internal, and health/metrics checks should target 9980. It is optional — if not deployed, admin and runtime servers continue operating normally. Webhook events accumulate in Redis with TTL until the service starts; evidence refs may be returned by the runtime before the signed envelope is available.

### Configuration

| Variable | Default | Description |
|---|---|---|
| `WEBHOOK_SECRET_ENCRYPTION_KEY` | required by default | AES-256-GCM key for signing secret encryption. Base64, 32 bytes. Same across all services. Generate: `openssl rand -base64 32`. Missing key fails admin/events startup unless `WEBHOOK_SECRET_ALLOW_PLAINTEXT=true` is explicitly set for local development. |
| `EVENT_TTL_DAYS` | 90 | Redis TTL for event records |
| `DELIVERY_TTL_DAYS` | 14 | Redis TTL for delivery records |
| `MAX_DELIVERY_AGE_MS` | 86400000 | Stale deliveries auto-fail after this age (24h default) |
| `dispatch.retry.poll-interval-ms` | 5000 | How often the retry scheduler scans for ready-to-retry deliveries. |
| `dispatch.retry.batch-size` | 100 | Max deliveries processed per retry-scan tick. |
| `dispatch.http.timeout-seconds` | 30 | HTTP request timeout per delivery attempt. |
| `dispatch.http.connect-timeout-seconds` | 5 | HTTP connect timeout per delivery attempt. |
| `EVIDENCE_SERVER_ID` | (empty) | CyclesEvidence issuer base including `/v1`. Blank disables evidence signing and leaves pending source records untouched. |
| `EVIDENCE_SIGNING_SIGNER_DID` | (empty) | Raw-hex public Ed25519 key. Must match the runtime server when evidence is enabled. |
| `EVIDENCE_SIGNING_PRIVATE_KEY_HEX` | (empty) | Raw-hex private Ed25519 key. Secret; set only on `cycles-server-events`. |

The per-subscription retry policy (exponential backoff) defaults to `max_retries=5`, `initial_delay_ms=1000`, `backoff_multiplier=2.0`, `max_delay_ms=60000`. A delivery older than `MAX_DELIVERY_AGE_MS` is failed immediately without further retries. See the [Events Service section in the Server Configuration Reference](/configuration/server-configuration-reference-for-cycles#events-service-configuration) for the full knob list.

### Running multiple instances

The Events Service is safe to run as multiple instances. `BLMOVE` moves a claimed job from `dispatch:pending` to the recoverable `dispatch:processing` list, and owner-token-checked acknowledgement prevents a stale worker from removing a successor's claim. A fleet-wide ordering lease currently serializes the claim/send critical section, so replicas provide failover rather than linear webhook throughput. Delivery semantics are at least once, not exactly once — webhook receivers should deduplicate on the event ID.

```yaml
cycles-events-1:
  image: ghcr.io/runcycles/cycles-server-events:0.1.25.25
  environment:
    REDIS_HOST: redis-primary
    REDIS_PORT: 6379
    REDIS_PASSWORD: ${REDIS_PASSWORD}
    WEBHOOK_SECRET_ENCRYPTION_KEY: ${WEBHOOK_SECRET_ENCRYPTION_KEY}

cycles-events-2:
  image: ghcr.io/runcycles/cycles-server-events:0.1.25.25
  environment:
    REDIS_HOST: redis-primary
    REDIS_PORT: 6379
    REDIS_PASSWORD: ${REDIS_PASSWORD}
    WEBHOOK_SECRET_ENCRYPTION_KEY: ${WEBHOOK_SECRET_ENCRYPTION_KEY}
```

### Events Service down

If the Events Service is unavailable:

1. Admin and runtime servers are **unaffected** — event dispatch and evidence source writes are fire-and-forget
2. Redis accumulates events with TTL (90-day events, 14-day deliveries)
3. On restart: stale deliveries older than `MAX_DELIVERY_AGE_MS` (default 24h) auto-fail; fresh ones deliver normally
4. If CyclesEvidence is enabled, `GET /v1/evidence/{id}` may return transient `404` until the events service signs and stores the envelope

## Network architecture

### Recommended topology

<NetworkTopology />

### Network isolation

- **Cycles Server** (port 7878): Accessible to your application. Can be on an internal network or behind an API gateway.
- **Admin Server** (port 7979): **Internal access only.** This manages tenants, API keys, and budgets. Never expose to the public internet.
- **Events Service** (app port 7980, management port 9980): **Internal access only.** Consumes from Redis and delivers webhooks outbound. Never needs inbound traffic from applications.
- **Redis** (port 6379): **Internal access only.** Never expose directly.

### TLS termination

Terminate TLS at the load balancer or API gateway. The Cycles Server itself runs plain HTTP. Example with nginx:

```nginx
server {
    listen 443 ssl;
    server_name cycles.internal.example.com;

    ssl_certificate /etc/ssl/certs/cycles.crt;
    ssl_certificate_key /etc/ssl/private/cycles.key;

    location / {
        proxy_pass http://cycles-server:7878;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}
```

## Capacity planning

### Rules of thumb

- **Redis memory:** ~1 KB per active reservation, ~500 bytes per budget ledger. 1 GB of Redis memory supports roughly 500K concurrent reservations.
- **Server CPU:** Each reservation involves 1 Redis Lua script execution (~1ms). A single server instance can handle thousands of reservations per second.
- **Latency:** Expect <5ms for reservation operations on a well-configured setup (server co-located with Redis).

### Scaling triggers

Add more **Cycles Server** instances when:
- Response latency exceeds 50ms at p99
- CPU utilization exceeds 70%

Add more **Events Service** instances when:
- The `dispatch:pending` queue depth grows consistently (`redis-cli LLEN dispatch:pending`)
- Webhook delivery latency exceeds acceptable thresholds
- Multiple instances are safe — each delivery job is claimed by one instance at a time via `BLMOVE`, with at-least-once delivery semantics

Scale **Redis** when:
- Memory utilization exceeds 80%
- Command latency exceeds 5ms

## Upgrade procedures

For the full preflight, service order, migration matrix, rollback boundaries, and verification checklist, use [Upgrading Cycles Safely](/how-to/upgrading-cycles).

### Rolling upgrade

All three services (Cycles Server, Admin Server, Events Service) are stateless — all state lives in Redis. You can do rolling upgrades with zero downtime:

1. Pull the new image: `docker pull ghcr.io/runcycles/cycles-server:NEW_VERSION`
2. Stop one instance at a time
3. Start the new version
4. Verify health check passes (`/actuator/health/readiness` on ports 7878, 7979, and events management port 9980)
5. Repeat for remaining instances

The Events Service can be upgraded independently. While it is down, webhook deliveries queue in Redis and are processed when the new version starts.

### Version compatibility

The Cycles protocol is versioned (`/v1`). Minor version upgrades (e.g., 0.1.23 → 0.1.24) are backward-compatible. Check the [changelog](/changelog) for breaking changes before major upgrades.

### Rollback

If an upgrade causes issues:

1. Stop the new version
2. Start the previous version
3. Redis state is compatible across minor versions

## Logging

### Log levels

Configure via Spring Boot:

```yaml
logging:
  level:
    io.runcycles: INFO      # Application logs
    org.springframework: WARN # Framework logs
```

Set `io.runcycles: DEBUG` for troubleshooting (includes full request/response logging).

### Structured logging

Add JSON logging for log aggregation systems:

```yaml
logging:
  pattern:
    console: '{"timestamp":"%d","level":"%p","logger":"%c","message":"%m"}%n'
```

Or use the Spring Boot JSON logging starter for full structured output.

## Operational runbooks

### Budget exhaustion alert

**Symptom:** Applications report `BUDGET_EXCEEDED` errors.

**Response:**
1. Check which scope is exhausted: `GET /v1/balances?tenant=...`
2. Determine if this is expected (legitimate traffic) or unexpected (runaway agent)
3. If expected: fund the budget via admin API (`POST .../fund` with `CREDIT`)
4. If unexpected: check active reservations for anomalies (`GET /v1/reservations?status=ACTIVE`)

### Reservation leak

**Symptom:** Budget `reserved` amount grows but `spent` stays flat. Reservations are being created but never committed or released.

**Response:**
1. List active reservations: `GET /v1/reservations?status=ACTIVE`
2. Check for reservations past their expected TTL
3. The expiry sweep should eventually clean these up. If it's not running, check the server logs.
4. Investigate the client application — it may be failing to commit or release.

### Commit failure after successful LLM call

**Symptom:** An LLM call (or other side-effecting action) completes successfully, but the subsequent commit to Cycles fails. The work happened and incurred real cost, but the budget ledger does not reflect it.

**Why this happens:**
- Transient network error between client and Cycles Server
- Cycles Server restart or Redis outage at commit time
- Client process crash after the LLM call but before commit

**What the lifecycle helpers do:**

The current Python, TypeScript, Spring Boot, and Rust lifecycle helpers journal known actual usage before the first settlement request. Transient and ambiguous failures retry with the original idempotency key. Retry exhaustion, authentication failure, unclassifiable 4xx responses, and process restart leave the record on disk. If the reservation expires before commit, replay switches the same durable record to `POST /v1/events`.

This guarantee begins only after the integration knows the actual amount and initiates settlement. A process crash before the provider returns usage cannot be recovered from SDK state alone; applications needing that stronger guarantee must durably checkpoint provider receipts or actual usage.

**Response:**
1. **Inspect the SDK journal and application logs.** Check for journal I/O failures, quarantined records, retained authentication failures, retry exhaustion, and event-fallback failures. The default base directory is `~/.runcycles/commit-journal`.
2. **Restore connectivity or credentials and restart/drain the client.** Replay uses the stored idempotency key. Configure the tenant explicitly so the journal partition remains discoverable after API-key rotation.
3. **Confirm ledger convergence:**
   ```bash
   curl -s "http://localhost:7979/v1/events" \
     -H "X-Cycles-API-Key: $API_KEY" \
     | jq '.events[] | select(.event_type == "event.applied") | {event_id, data}'
   ```

Do not create a second reconciliation event with a fresh key while a journal record is pending: if the original ambiguous request succeeded, that can double-record spend. Manual reconciliation is appropriate only after proving that no durable record or successful same-key settlement exists.

**Prevention:**
- Keep journaling enabled and place its directory on a persistent volume rather than an ephemeral container layer
- Configure a stable tenant identity so pending records survive API-key rotation
- Keep retry enabled for prompt convergence; disabling retry does not disable journal durability
- Allow the SDK's bounded shutdown drain to run, and preserve records when the drain times out
- Alert on pending-record age/count, journal failures, heartbeat stop messages, and expired reservation rate (see [Monitoring and Alerting](/how-to/monitoring-and-alerting))

### Redis connection loss

**Symptom:** All reservation operations fail with 500 errors. Events Service also stops processing deliveries.

**Response:**
1. Check Redis connectivity: `redis-cli ping`
2. Check server logs for connection errors on all three services (ports 7878, 7979, 7980)
3. Restart services if Redis connection pool is exhausted
4. Active reservations with remaining TTL are preserved in Redis and will resume when connectivity returns
5. Queued webhook deliveries resume automatically when the Events Service reconnects

### Webhook delivery failures

**Symptom:** Webhook endpoints are not receiving events. Queue depth grows.

**Response:**
1. Check Events Service health: `GET http://localhost:9980/actuator/health/readiness`
2. Check queue depth: `redis-cli LLEN dispatch:pending`
3. Check if subscription was auto-disabled: `GET /v1/admin/webhooks/{subscription_id}`
4. Re-enable if needed: `PATCH /v1/admin/webhooks/{subscription_id}` with `{"status": "ACTIVE"}`
5. Verify `WEBHOOK_SECRET_ENCRYPTION_KEY` matches across all services

## Next steps

- [Webhook Integrations](/how-to/webhook-integrations) — PagerDuty, Slack, ServiceNow examples
- [Client Performance Tuning](/how-to/client-performance-tuning) — timeout, retry, and connection pool optimization
- [Security Hardening](/how-to/security-hardening) — Redis AUTH, TLS, key rotation, webhook security
- [Monitoring and Alerting](/how-to/monitoring-and-alerting) — metrics and alerting setup
- [Server Configuration Reference](/configuration/server-configuration-reference-for-cycles) — all configuration properties
- [Redis Backup, Restore, and Disaster Recovery](/how-to/redis-backup-restore-disaster-recovery) — backup validation and full-dataset recovery
- [Upgrading Cycles Safely](/how-to/upgrading-cycles) — rolling order, migrations, rollback, and verification



# Prometheus Metrics Reference

This page enumerates every custom metric that Cycles' reference servers expose on their Prometheus endpoints, along with tag definitions, cardinality guidance, and a sample scrape config. For the higher-level observability playbook (alert recipes, SLOs) see [Monitoring and Alerting](/how-to/monitoring-and-alerting).

Cycles' Micrometer instrumentation uses dotted source names (`cycles.*`) which Prometheus rewrites to underscores with a `_total` suffix on scrape. Source names below are the raw Micrometer identifier; the **Prometheus** column is what you actually query and alert on.

::: info Tenant-tag cardinality flag
Every counter tagged with `tenant` respects a per-service toggle, `cycles.metrics.tenant-tag.enabled` (env `CYCLES_METRICS_TENANT_TAG_ENABLED`) — but the **defaults differ by service**: `cycles-server` (runtime) defaults to `true`, while `cycles-server-events` defaults to `false`. The admin server's `cycles_admin_*` counters carry no `tenant` tag at all, so the flag doesn't apply there. Deployments with many thousands of tenants can flip the runtime flag to `false` to drop the per-tenant series and keep Prometheus cardinality bounded; deployments that want per-tenant webhook drill-downs must explicitly enable it on the events service. Set the flag consistently across the two services so dashboards can share the same tag schema.

For the operation counters below, null or blank tag values are normalised to the sentinel `UNKNOWN` (exception: `cycles.evidence.emit_failed` uses lowercase `unknown` for a null `artifact_type` and does not normalise blanks). Missing tags would otherwise collapse series — making it look like traffic moved when the upstream data actually just got sparse.
:::

## Scrape targets

| Service | Scrape port | Prometheus path |
|---|---|---|
| `cycles-server` (runtime) | `7878` (same as API) | `/actuator/prometheus` |
| `cycles-server-events` (dispatcher) | **`9980`** (dedicated management port, split from app port `7980` in v0.1.25.9, env `MANAGEMENT_PORT`) | `/actuator/prometheus` |
| `cycles-server-admin` | `7979` (same as API) | `/actuator/prometheus` |

**Events-service port split.** Starting with `cycles-server-events` v0.1.25.9, the `health`, `info`, and `prometheus` actuator endpoints moved from the application port `7980` to a dedicated management port (default `9980`, env `MANAGEMENT_PORT`). Scrape configs, kubelet probes, and Docker `HEALTHCHECK` commands must target `:9980` — the published Docker image's `HEALTHCHECK` is already updated. Do not publish `7980` for webhook delivery; keep both events-service ports internal-only.

## Runtime server (`cycles-server`)

Introduced in v0.1.25.10 and expanded in later v0.1.25 releases. The current runtime inventory is 11 counters and one timer under the `cycles.*` namespace.

| Source | Prometheus | Type | Tags | Description |
|---|---|---|---|---|
| `cycles.reservations.reserve` | `cycles_reservations_reserve_total` | Counter | `tenant`, `decision`, `reason`, `overage_policy` | Every `POST /v1/reservations` outcome. |
| `cycles.reservations.commit` | `cycles_reservations_commit_total` | Counter | `tenant`, `decision`, `reason`, `overage_policy` | Every `POST /v1/reservations/{id}/commit` outcome. |
| `cycles.reservations.release` | `cycles_reservations_release_total` | Counter | `tenant`, `actor_type`, `decision`, `reason` | Every release outcome (`decision` is `RELEASED` or `DENY`). `actor_type` distinguishes tenant-driven releases from v0.1.25.8 admin-on-behalf-of releases. |
| `cycles.reservations.extend` | `cycles_reservations_extend_total` | Counter | `tenant`, `decision`, `reason` | Every `POST /v1/reservations/{id}/extend` outcome. |
| `cycles.reservations.expired` | `cycles_reservations_expired_total` | Counter | `tenant` | Each reservation the expiry sweep actually marks EXPIRED. Skipped reservations (still in grace, already finalised) do not increment. |
| `cycles.reservations.quarantined` | `cycles_reservations_quarantined_total` | Counter | `tenant`, `reason` | Malformed reservation records quarantined by maintenance. `reason`: `INVALID_EXPIRY`, `INVALID_ESTIMATE`, `MISSING_SCOPES`, or `MALFORMED_SCOPES`. |
| `cycles.reservations.created_at_index.reads` | `cycles_reservations_created_at_index_reads_total` | Counter | `outcome` | Read-path result for the optional created-at sorted index. `outcome`: `INDEX`, `SCAN_DISABLED`, `SCAN_NOT_READY`, `SCAN_DRIFT`, or `SCAN_ERROR`. |
| `cycles.maintenance.runs` | `cycles_maintenance_runs_total` | Counter | `job`, `outcome` | Scheduled maintenance run outcomes. |
| `cycles.maintenance.duration` | `cycles_maintenance_duration_seconds` | Timer | `job`, `outcome` | Duration of scheduled maintenance runs. |
| `cycles.events` | `cycles_events_total` | Counter | `tenant`, `decision`, `reason`, `overage_policy` | Every `POST /v1/events` outcome. |
| `cycles.overdraft.incurred` | `cycles_overdraft_incurred_total` | Counter | `tenant` | Any commit or event that actually accrued non-zero debt. Unit-free signal — debt amount is tracked by the balance store, not here, to avoid leaking user-value distributions into metrics. |
| `cycles.evidence.emit_failed` | `cycles_evidence_emit_failed_total` | Counter | `artifact_type` | Evidence-source enqueue failures (fail-open path) — a lifecycle op committed but its evidence record could not be queued (e.g. Redis died just after the ledger write). No `tenant` tag. |

### Tag value reference (runtime)

| Tag | Values |
|---|---|
| `decision` | Per-operation success/deny outcome. Reserve: `ALLOW`, `ALLOW_WITH_CAPS`, `DENY`. Commit: `COMMITTED`, `DENY`. Release: `RELEASED`, `DENY`. Extend: `ACTIVE`, `DENY`. Events: `APPLIED`, `DENY`. |
| `reason` | `OK` on success, `IDEMPOTENT_REPLAY` on idempotent replays, an `ErrorCode` name on denials (`BUDGET_EXCEEDED`, `BUDGET_FROZEN`, `BUDGET_CLOSED`, `RESERVATION_EXPIRED`, `RESERVATION_FINALIZED`, `IDEMPOTENCY_MISMATCH`, `UNIT_MISMATCH`, `OVERDRAFT_LIMIT_EXCEEDED`, `DEBT_OUTSTANDING`, `MAX_EXTENSIONS_EXCEEDED`, `NOT_FOUND`, …), or `INTERNAL_ERROR` on unexpected failures. `UNKNOWN` when the code path doesn't produce one. |
| `overage_policy` | `REJECT`, `ALLOW_IF_AVAILABLE`, `ALLOW_WITH_OVERDRAFT` — which commit overage policy was in effect for the scope. `UNKNOWN` when not resolved. |
| `actor_type` | `tenant` (tenant-driven) or `admin_on_behalf_of` (admin-driven, v0.1.25.8+). |
| `job` | `reservation_expiry`, `audit_retention`, `event_retention`, `created_at_repair`, or `created_at_sweep`. |
| `outcome` (maintenance) | `success`, `failed`, `skipped_locked`, `skipped_disabled`, `lease_error`, or `lease_lost`. |

### Not instrumented (by design)

- **HTTP-layer latency histograms** — Spring Boot auto-emits `http.server.requests` with `uri` / `method` / `status` labels already. Use those for per-endpoint latency.
- **Lua-script execution time** — `EVALSHA` timings would largely duplicate the HTTP timer for request-synchronous scripts. The expiry-sweep counter (`cycles.reservations.expired`) covers the one non-request-driven path.

## Events service (`cycles-server-events`)

Introduced in v0.1.25.6 and expanded with evidence, boundary, dispatcher, and security instrumentation. The current inventory is 17 counters and one timer. `cycles.webhook.*` rewrites to `cycles_webhook_*_total` (counters) or `cycles_webhook_*_seconds` (timer). This service emits an explicit latency timer because it is the HTTP *client*—Spring's auto `http.server.requests` does not cover its primary I/O surface.

| Source | Prometheus | Type | Tags | Description |
|---|---|---|---|---|
| `cycles.webhook.delivery.attempts` | `cycles_webhook_delivery_attempts_total` | Counter | `tenant`, `event_type` | Every outbound delivery attempt (first attempt + every retry). |
| `cycles.webhook.delivery.success` | `cycles_webhook_delivery_success_total` | Counter | `tenant`, `event_type`, `status_code_family` | Successful deliveries. `status_code_family`: `2xx`. |
| `cycles.webhook.delivery.failed` | `cycles_webhook_delivery_failed_total` | Counter | `tenant`, `event_type`, `reason` | Failed deliveries. `reason` is one of `event_not_found`, `subscription_not_found`, `subscription_inactive`, `http_4xx`, `http_5xx`, `transport_error`, `ssrf_blocked`. |
| `cycles.webhook.delivery.retried` | `cycles_webhook_delivery_retried_total` | Counter | `tenant`, `event_type` | Deliveries that re-entered the retry queue. |
| `cycles.webhook.delivery.stale` | `cycles_webhook_delivery_stale_total` | Counter | `tenant` | Deliveries auto-failed on pickup for exceeding `dispatch.max-delivery-age-ms` (default 24h). |
| `cycles.webhook.delivery.dead_lettered` | `cycles_webhook_delivery_dead_lettered_total` | Counter | `reason` | Delivery jobs moved to dead letter. |
| `cycles.webhook.delivery.boundary_skipped` | `cycles_webhook_delivery_boundary_skipped_total` | Counter | `tenant`, `event_type`, `category` | Deliveries intentionally skipped at a configured dispatch boundary. |
| `cycles.webhook.subscription.auto_disabled` | `cycles_webhook_subscription_auto_disabled_total` | Counter | `tenant`, `reason` | Subscriptions auto-disabled after consecutive failures crossed the threshold. `reason` is `consecutive_failures` (the accompanying `webhook.disabled` Event's payload uses the longer `disable_reason=consecutive_failures_exceeded_threshold`). Always emitted together with a `webhook.disabled` Event (v0.1.25.11). |
| `cycles.webhook.delivery.latency` | `cycles_webhook_delivery_latency_seconds` | Timer | `tenant`, `event_type`, `outcome` | Round-trip time on deliveries that actually produced a transport response. `outcome`: `success` or `failure`. Upstream failures (event_not_found, etc.) have no meaningful latency and do not record to this timer. |
| `cycles.webhook.events.payload.invalid` | `cycles_webhook_events_payload_invalid_total` | Counter | `type`, `rule` | Non-fatal shape discrepancy found by `EventPayloadValidator` on an ingested event. No tenant dimension — the discrepancy is about payload shape, not tenant traffic. `rule` values: `missing_required`, `unknown_event_type`, `unknown_category`, `category_mismatch`, `budget_data_shape`, `reset_spent_shape`, `trace_id_shape`. |
| `cycles.evidence.claimed` | `cycles_evidence_claimed_total` | Counter | `artifact_type` | Evidence-source records claimed for processing. |
| `cycles.evidence.stored` | `cycles_evidence_stored_total` | Counter | `artifact_type` | Signed evidence envelopes stored successfully. |
| `cycles.evidence.dead_lettered` | `cycles_evidence_dead_lettered_total` | Counter | `artifact_type`, `reason` | Evidence-source records moved to dead letter. |
| `cycles.evidence.retry_deferred` | `cycles_evidence_retry_deferred_total` | Counter | `artifact_type`, `reason` | Evidence work deferred for a later retry. |
| `cycles.webhook.dispatcher.event.published` | `cycles_webhook_dispatcher_event_published_total` | Counter | `event_type` | Dispatcher lifecycle events published successfully. |
| `cycles.webhook.dispatcher.event.deferred` | `cycles_webhook_dispatcher_event_deferred_total` | Counter | `event_type`, `reason` | Dispatcher lifecycle events deferred. |
| `cycles.webhook.dispatcher.event.dead_lettered` | `cycles_webhook_dispatcher_event_dead_lettered_total` | Counter | `event_type`, `reason` | Dispatcher lifecycle events moved to dead letter. |
| `cycles.webhook.security.config.indeterminate` | `cycles_webhook_security_config_indeterminate_total` | Counter | — | Delivery attempts blocked because the webhook-security configuration could not be determined safely. |

### Tag value reference (events)

| Tag | Values |
|---|---|
| `event_type` | Event kind from the [Event Payloads Reference](/protocol/event-payloads-reference) (e.g. `reservation.denied`, `budget.exhausted`, `webhook.disabled`). Up to 51 registered values, plus additive implementation events over time. |
| `status_code_family` | `2xx` (success bucket). Non-2xx responses land on `cycles_webhook_delivery_failed_total` with `reason` instead. |
| `reason` (on `_failed_total`) | `event_not_found`, `subscription_not_found`, `subscription_inactive`, `http_4xx`, `http_5xx`, `transport_error` (timeouts, connection resets, DNS/SSL failures — status code 0), `ssrf_blocked`. |
| `artifact_type` | `decide`, `reserve`, `commit`, `release`, `error`, or `UNKNOWN`. |
| `event_type` (dispatcher lifecycle) | `webhook.disabled`, `system.webhook_delivery_failed`, or `UNKNOWN`. |

## Admin server (`cycles-server-admin`)

Exposed since admin observability rollout (v0.1.25.9+). Metric names use the `cycles_admin_*` prefix.

| Prometheus | Type | Tags | Description |
|---|---|---|---|
| `cycles_admin_audit_writes_total` | Counter | `path_class`, `outcome` | Audit-trail write accounting. `outcome` values: `written`, `error` (Redis write failed — alert on nonzero), `sampled-out` (pre-auth sampling dropped the entry per `audit.sample.unauthenticated`). `path_class` groups endpoints for coarse-grained triage. Shipped v0.1.25.20 alongside the audit-on-failure coverage. |
| `cycles_admin_events_emitted_total` | Counter | `type`, `result` | Admin-emitted Event accounting. `result`: `success` or `failure`. |
| `cycles_admin_events_payload_invalid_total` | Counter | `type`, `expected_class` | Jackson round-trip found an Event payload that didn't match its declared schema. Non-fatal — admin continues to accept the event. Shipped v0.1.25.12. |
| `cycles_admin_webhook_dispatched_total` | Counter | `result` | Enqueue-to-dispatcher accounting. `result`: `queued`, `failure`, or `boundary_skipped`. The end-to-end delivery metric is `cycles_webhook_delivery_*` on the events service. |
| `cycles_admin_tenant_close_outbox_dead_letter_total` | Counter | `resource_type` | Tenant-close outbox records moved to dead letter. |
| `cycles_admin_tenant_close_reconcile_incomplete_total` | Counter | — | Tenant-close reconciliation runs that ended with incomplete work. |
| `cycles_admin_tenant_close_reconcile_errors_total` | Counter | — | Tenant-close reconciliation errors. |

## Sample scrape config

Since `cycles-server` 0.1.25.45, the runtime and admin `/actuator/prometheus` endpoints require the `X-Admin-API-Key` header (liveness/readiness probe paths stay public). Prometheus v3.0+ can send it via `http_headers`; on older versions, front the endpoint with a header-injecting proxy. The events service's management port has no auth filter — keep it internal-only.

```yaml
scrape_configs:
  - job_name: cycles-runtime
    metrics_path: /actuator/prometheus
    http_headers:
      X-Admin-API-Key:
        secrets: ['${ADMIN_API_KEY}']
    static_configs:
      - targets: ['cycles-server:7878']

  - job_name: cycles-events
    metrics_path: /actuator/prometheus   # no admin key needed — network-restrict instead
    static_configs:
      - targets: ['cycles-server-events:9980']  # management port, NOT the app port (7980)

  - job_name: cycles-admin
    metrics_path: /actuator/prometheus
    http_headers:
      X-Admin-API-Key:
        secrets: ['${ADMIN_API_KEY}']
    static_configs:
      - targets: ['cycles-server-admin:7979']
```

## Cardinality guidance

The `tenant` tag is the dominant cardinality driver. It appears on eight runtime counters and six events-service metrics in the current inventory; each metric can expand further across its bounded decision, reason, event, or outcome labels. If Prometheus memory or scrape duration becomes a concern:

1. **Flip `cycles.metrics.tenant-tag.enabled` to `false`** on the runtime server (the events service already defaults to `false`). Counters drop the `tenant` tag; you lose per-tenant drill-downs but keep decision / reason / outcome signals.
2. **Aggregate at scrape time** with `metric_relabel_configs` to drop the tag selectively on high-cardinality metrics while keeping it on the ones you still want tenant-sliced.
3. **Keep per-tenant on Timer, drop on Counters** if delivery-latency-per-tenant is the signal you care about most.

`event_type` is bounded by the spec (51 registered values today, additive over time). `reason` and `decision` are enum-bounded and safe.

## Quick alert recipes

| Signal | Query | Typical threshold |
|---|---|---|
| Audit-write error | `sum(rate(cycles_admin_audit_writes_total{outcome="error"}[5m])) > 0` | Any nonzero — audit trail has a gap. |
| Webhook auto-disable rate | `sum(rate(cycles_webhook_subscription_auto_disabled_total[15m])) > 0` | Any nonzero — a subscription was just auto-disabled. |
| Overdraft rate spike | `sum(rate(cycles_overdraft_incurred_total[5m])) / sum(rate(cycles_reservations_commit_total[5m])) > 0.05` | Over 5% of commits are going into overdraft. |
| Dispatch worst-case latency | `max_over_time(cycles_webhook_delivery_latency_seconds_max{outcome="success"}[5m])` | Over 10s — something downstream is struggling. (For a true p95 via `histogram_quantile` over `_bucket` series, first enable Micrometer percentile histograms for this timer — no service publishes `_bucket` series by default.) |
| Stale-delivery rate | `sum(rate(cycles_webhook_delivery_stale_total[1h])) > 0` | Any nonzero — deliveries are sitting in-queue past `dispatch.max-delivery-age-ms`. |

For the full set of alerts and SLOs see [Monitoring and Alerting](/how-to/monitoring-and-alerting) and [Production Operations](/how-to/production-operations-guide).



# Redis Backup, Restore, and Disaster Recovery

Redis is the durable state store for the Cycles runtime, admin, and events services. A recovery plan must restore the complete Redis dataset as one consistency boundary—not selected keys.

A Cycles dataset includes budget balances, reservations, idempotency records, tenants, API keys, policies, webhook subscriptions, events, audit records, delivery queues, and CyclesEvidence queues and envelopes. Restoring only one family can separate a balance from the reservations or idempotency records that explain it.

This runbook targets Redis Open Source 7+, the minimum supported Redis generation for Cycles. For Redis Cloud or Redis Software, use the provider's backup and restore controls, then apply the Cycles validation and cutover steps below.

## Choose recovery objectives

Set and test these objectives before an incident:

| Objective | Decision |
|---|---|
| Recovery point objective (RPO) | Maximum Cycles writes you can lose. AOF `everysec` generally targets about one second; periodic RDB alone can lose the interval since the last snapshot. |
| Recovery time objective (RTO) | Maximum time to restore Redis, validate Cycles state, and reopen writes. Measure this with production-sized restore drills. |
| Backup retention | Keep multiple hourly and daily restore points, not only the newest file. |
| Offsite boundary | Store at least one encrypted copy outside the Redis host and failure domain. |
| Restore owner | Name the operator authorized to stop writes, select a restore point, and approve cutover. |

Redis documents the durability and performance tradeoffs in its [persistence guide](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/). For production Cycles state, use both AOF and periodic RDB snapshots unless your managed Redis service provides an equivalent durability model.

## Production persistence baseline

```conf
appendonly yes
appendfsync everysec

save 900 1
save 300 10

maxmemory-policy noeviction
stop-writes-on-bgsave-error yes
```

`noeviction` is required for budget integrity. Evicting a ledger, reservation, or idempotency record can make later decisions inconsistent.

RDB is a compact point-in-time backup. AOF replays writes and offers a tighter recovery point. When both are enabled, Redis uses AOF on restart because it is normally the more complete dataset.

## Inventory the live instance

Record the persistence paths and health before every backup:

```bash
redis-cli CONFIG GET dir
redis-cli CONFIG GET dbfilename
redis-cli CONFIG GET appenddirname
redis-cli INFO persistence
redis-cli DBSIZE
```

Use your platform's secret injection for Redis authentication; avoid putting passwords directly in command history or process arguments.

Capture:

- Redis version and deployment topology;
- logical database number used by Cycles;
- `DBSIZE`;
- `rdb_last_bgsave_status`;
- `aof_enabled`, `aof_last_write_status`, and `aof_last_bgrewrite_status`;
- queue depths for `dispatch:pending`, `dispatch:processing`, `dispatch:retry`, `evidence:pending`, and `evidence:processing`;
- the Cycles component versions and encryption/evidence identity configuration.

Key counts are a coarse comparison only. TTL expiry and retention jobs can legitimately change them between backup and restore.

## Create an RDB backup

1. Start a background snapshot:

   ```bash
   redis-cli BGSAVE
   ```

2. Poll `INFO persistence` until:

   ```text
   rdb_bgsave_in_progress:0
   rdb_last_bgsave_status:ok
   ```

3. Record `LASTSAVE`, then copy the completed RDB file from the configured `dir` and `dbfilename`.
4. Compute a cryptographic checksum, encrypt the backup, and copy it off-host.
5. Record the checksum, Redis version, creation time, key count, and Cycles fleet versions alongside the backup.

Redis writes RDB snapshots to a temporary file and atomically replaces the completed snapshot, so copying the completed RDB while Redis remains online is safe. `BGSAVE` behavior and `LASTSAVE` verification are documented in the official [`BGSAVE` reference](https://redis.io/docs/latest/commands/bgsave/).

## Create an AOF backup on Redis 7+

Redis 7 stores AOF as a multipart set with a manifest in `appenddirname`. Copy the entire directory, not one file.

1. Read and record the current `auto-aof-rewrite-percentage`.
2. Temporarily disable automatic rewrites:

   ```bash
   redis-cli CONFIG SET auto-aof-rewrite-percentage 0
   ```

3. Check `INFO persistence` until `aof_rewrite_in_progress:0`. Do not run `BGREWRITEAOF` during the copy.
4. Copy or archive the complete configured `appenddirname`, including its manifest.
5. Restore the previous rewrite percentage immediately:

   ```bash
   redis-cli CONFIG SET auto-aof-rewrite-percentage 100
   ```

   Replace `100` with the value recorded in step 1.

6. Encrypt, checksum, and transfer the archive off-host.

The temporary rewrite pause is required because copying multipart AOF files during a rewrite can produce an invalid backup. Redis's [AOF backup procedure](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/#backing-up-aof-persistence) is authoritative.

## Validate every backup

A successful file copy is not a successful recovery test.

1. Restore the backup into an isolated Redis instance using the same Redis major version and configuration shape.
2. Keep the isolated instance unreachable from all Cycles services and clients.
3. Confirm Redis starts without persistence-load errors.
4. Compare `INFO persistence`, `DBSIZE`, and representative key types with the backup manifest.
5. Point isolated runtime and admin instances at a disposable copy of the restored Redis and perform the read-only checks in [Validate Cycles state](#_4-validate-cycles-state). Test the events worker only with outbound network access blocked: it is an active queue consumer, not a read-only service.
6. Record restore duration, data checks, and any manual steps. The measured duration—not the archive copy time—is your practical RTO.

Run this drill on a schedule and after changes to Redis version, persistence mode, encryption keys, or Cycles storage behavior.

## Restore after data loss

### 1. Declare a write outage

Stop or isolate all writers before restoring:

- runtime server instances;
- admin server instances;
- events-service workers;
- automation calling the runtime or admin APIs.

Stopping only the dashboard is insufficient. SDKs, agents, webhooks, and background jobs can still write.

### 2. Preserve incident evidence

Before replacing anything:

- snapshot or archive the failed Redis volume if it is readable;
- save Redis and Cycles logs;
- record the failure time and last known good operation;
- record checksums for the candidate backup;
- do not run `redis-check-aof --fix` on the only copy.

For AOF corruption, Redis recommends making a copy and running `redis-check-aof` without `--fix` first. Repair can discard data from the corruption point onward.

### 3. Restore into a new Redis instance

Prefer a new isolated instance or volume over overwriting the failed one.

- For RDB recovery, place the snapshot at the configured `dir`/`dbfilename` before Redis starts.
- For Redis 7 multipart AOF recovery, restore the complete `appenddirname` and manifest.
- Preserve file ownership and permissions required by the Redis process.
- Start Redis with the intended production persistence settings.

Do not combine an RDB from one restore point with AOF files from another. If both are present, Redis loads AOF, so an unintended stale AOF directory can silently override the RDB you meant to test.

### 4. Validate Cycles state

Keep application writes blocked while validating:

```bash
redis-cli PING
redis-cli INFO persistence
redis-cli DBSIZE
redis-cli LLEN dispatch:pending
redis-cli LLEN dispatch:processing
redis-cli ZCARD dispatch:retry
redis-cli LLEN evidence:pending
redis-cli LLEN evidence:processing
```

Then start one isolated runtime instance and one admin instance against the restored Redis. Test the events service against a disposable clone with outbound webhook access blocked; starting it against the cutover candidate can consume queues and change delivery state.

Verify:

- runtime and admin readiness report `UP`; the isolated events-worker test also reports `UP`;
- tenant, budget, API-key metadata, policy, webhook, event, and audit list endpoints can read representative records;
- balances reconcile with expected allocations, spent amounts, reserved amounts, debt, and remaining values;
- open reservations refer to existing ledgers and have plausible expirations;
- failed and pending delivery/evidence queues have expected depths;
- encrypted webhook secrets can be decrypted using the original `WEBHOOK_SECRET_ENCRYPTION_KEY`;
- the evidence signer identity and retired-key history match the restored evidence records.

Use read-only requests until the restore point has been approved.

### 5. Account for the recovery window

A point-in-time restore can remove writes that clients observed as successful after the restore point. It can also restore an idempotency record whose downstream side effect occurred after that point.

Before reopening writes:

- identify runtime/admin requests accepted between the backup time and the outage;
- reconcile downstream LLM, tool, or payment-side effects with Cycles reservations and commits;
- do not blindly replay mutating requests with new idempotency keys;
- expect already-expired Redis TTL records to disappear during or immediately after load;
- decide whether webhook/evidence queue items need replay, suppression, or downstream deduplication.

Cycles records economic exposure; it does not reverse downstream actions when Redis is restored.

### 6. Cut over

1. Put the restored Redis behind the production endpoint.
2. Start one instance of each Cycles service.
3. Repeat the read-only validation.
4. Permit one controlled reserve → commit lifecycle and confirm the balance delta.
5. Start remaining service replicas.
6. Re-enable client traffic gradually while watching errors, denials, queue depth, and Redis persistence status.

## Roll back the restore

Keep the pre-restore Redis volume and restored candidate until validation is complete. If the candidate fails:

1. block Cycles writes again;
2. preserve the candidate and its logs;
3. switch back to the previous isolated instance or choose an earlier verified restore point;
4. repeat validation before reopening traffic.

Never switch between two independently writable Redis instances. Cycles does not merge divergent ledgers.

## Drill checklist

- [ ] RPO and RTO are documented and approved.
- [ ] RDB and multipart AOF backups are encrypted and stored off-host.
- [ ] Backup checksums and version metadata are recorded.
- [ ] A production-sized restore has been completed in isolation.
- [ ] The complete Cycles state—not selected key families—was restored.
- [ ] Webhook encryption and evidence signer identities were available during the drill.
- [ ] Read-only Cycles validation passed.
- [ ] One controlled reserve → commit lifecycle passed before reopening traffic.
- [ ] Measured restore and validation time meets the RTO.
- [ ] Reconciliation ownership for the recovery window is assigned.

## Related

- [Redis persistence](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/) — authoritative RDB and AOF behavior
- [Production Operations](/how-to/production-operations-guide)
- [Server Configuration Reference](/configuration/server-configuration-reference-for-cycles)
- [Monitoring and Alerting](/how-to/monitoring-and-alerting)
- [Retry Storms and Idempotency Failures](/incidents/retry-storms-and-idempotency-failures)



# Rolling Over Billing Periods with RESET_SPENT

Budget ledgers in Cycles track five counters: `allocated`, `spent`, `reserved`, `debt`, and `remaining` (`remaining = allocated − spent − reserved − debt`). At the end of a billing period — monthly, weekly, or whatever your plan defines — you typically want to restate the *allocation* and clear the *spend*, while preserving in-flight reservations and any debt that should persist into the next period.

That is exactly what `RESET_SPENT` does. It was added in `cycles-server-admin` v0.1.25.18 as a narrower alternative to the existing `RESET` operation, and is available as a funding operation on `POST /v1/admin/budgets/fund`.

## RESET vs RESET_SPENT

The two operations are easy to confuse. Here is how they differ:

| Operation | Sets `allocated` | Clears `spent` | Preserves `reserved` | Preserves `debt` |
|-----------|------------------|----------------|----------------------|------------------|
| `RESET` | **Yes** — to the `amount` in the request | No — preserved | Yes | Yes |
| `RESET_SPENT` | **Yes** — to the `amount` in the request (**required**) | **Yes** — to zero, or to the optional `spent` value | Yes | Yes |

- **`RESET`** changes the size of the budget. The allocation counter is rewritten to whatever you passed in. `spent` carries over. Use this when a customer upgrades or downgrades mid-period.
- **`RESET_SPENT`** starts a new billing period. `allocated` is set to the (required) `amount` in the request — pass the current allocation if the ceiling is not changing — and the `spent` counter is zeroed out (or set to the optional `spent` value, e.g., for a prorated correction).

The protocol was missing a way to clear `spent` at a period boundary: `RESET` resizes the allocation but preserves spend, and issuing corrective `DEBIT`s was error-prone. `RESET_SPENT` closes that gap with a single operation that sets the new period's allocation and resets the spend.

## Basic monthly rollover

The most common case — a cron job that runs at the start of each billing period and zeroes out spend while restating the allocation. `amount` is required and becomes the new `allocated` — pass the current allocation to keep the ceiling unchanged. When authenticating with the admin key, `tenant_id` is also required:

```bash
curl -X POST "http://localhost:7979/v1/admin/budgets/fund?tenant_id=acme-corp&scope=tenant:acme-corp&unit=USD_MICROCENTS" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "RESET_SPENT",
    "amount": { "amount": 100000000, "unit": "USD_MICROCENTS" },
    "idempotency_key": "rollover-acme-2026-05",
    "reason": "Monthly rollover for billing period 2026-05"
  }'
```

After this call:

- `allocated` — set to the `amount` in the request (`100000000` here).
- `spent` — zero.
- `reserved` — unchanged. Any reservations that were live at the moment of the call continue to hold their budget, and commit normally.
- `debt` — unchanged.
- `remaining` — recomputed as `allocated - spent - reserved - debt`.

The idempotency key should encode the tenant and the period being started. If the cron retries, the replay returns the original response and the counters do not move twice.

## Prorated corrections

If a customer upgrades mid-period and you need to credit back some of the spend they incurred on the old plan, you can pass an explicit `spent` value instead of clearing to zero:

```bash
curl -X POST "http://localhost:7979/v1/admin/budgets/fund?tenant_id=acme-corp&scope=tenant:acme-corp&unit=USD_MICROCENTS" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "RESET_SPENT",
    "amount": { "amount": 100000000, "unit": "USD_MICROCENTS" },
    "idempotency_key": "prorate-acme-2026-04-17",
    "spent": { "amount": 3200000, "unit": "USD_MICROCENTS" },
    "reason": "Prorated spend after mid-period plan change"
  }'
```

This sets `spent` to exactly `3200000` microcents rather than zero. Use this pattern for:

- Plan changes where the carry-over spend is recalculated
- Refunds issued as a spend reduction rather than an allocation increase
- Migration from a legacy billing system where opening balances are non-zero

## Why reserved and debt are preserved

This is deliberate and matches how production systems actually roll over:

- **Reserved budget represents in-flight work.** An agent that started a reservation at 23:59:58 is still executing at 00:00:02. Zeroing `reserved` would cause its commit to double-count (the commit would subtract from the fresh period's budget while the reservation's hold was already released). Preserving `reserved` lets the existing reservation commit cleanly.
- **Debt represents money you've already let the tenant spend past the cap.** If the old period ended in overdraft, that debt is a real liability. It should either roll forward (the default) or be repaid explicitly with `REPAY_DEBT`. Silently clearing debt at rollover would erase the accounting.

If you want to explicitly zero out reservations or debt, use the targeted operations (`POST /v1/reservations/{id}/release` or a `REPAY_DEBT` funding call) alongside the `RESET_SPENT` call.

## Events emitted

A successful `RESET_SPENT` emits `budget.reset_spent` (v0.1.25.18+). The payload carries `previous_state` and `new_state` snapshots — including the pre-rollover `spent` — plus a `spent_override_provided` flag that distinguishes routine rollovers (`false`, spent defaulted to 0) from explicit adjustments (`true`, the request supplied a `spent` value). Useful for downstream billing systems that want to archive the period's total on the event stream rather than polling balances.

```json
{
  "event_type": "budget.reset_spent",
  "data": {
    "ledger_id": "led_acme_default",
    "scope": "tenant:acme-corp",
    "unit": "USD_MICROCENTS",
    "operation": "RESET_SPENT",
    "previous_state": {
      "allocated": 100000000,
      "spent": 87340000,
      "reserved": 1200000,
      "debt": 0,
      "remaining": 11460000
    },
    "new_state": {
      "allocated": 100000000,
      "spent": 0,
      "reserved": 1200000,
      "debt": 0,
      "remaining": 98800000
    },
    "spent_override_provided": false,
    "reason": "Monthly rollover for billing period 2026-05"
  }
}
```

See the [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol) for the full event envelope.

## Scheduling the rollover

Budget ledgers carry declarative period metadata — `rollover_policy` (`NONE`, `CARRY_FORWARD`, `CAP_AT_ALLOCATED`) plus `period_start`/`period_end` — that describes how unused budget should be handled at period boundaries. But the server does not execute those boundaries on a schedule: there is no built-in cron, and `RESET_SPENT` is the explicit operation you call to effect the rollover. You trigger it however fits your operational model:

- **External cron.** A scheduled job that reads a list of active tenants from your own tenancy database and calls `RESET_SPENT` for each on the first of the month.
- **Stripe webhook-driven.** A handler for Stripe's `invoice.finalized` event that rolls over the corresponding tenant as part of invoice reconciliation.
- **Balance-driven.** Poll balances and apply your own plan rules when utilization changes. Do not use exhaustion alone as a billing-period boundary: it can happen before the calendar or invoice period ends, and the current runtime does not emit configurable pre-exhaustion threshold events.

In every case, make the idempotency key include the target period, so a retry or duplicate trigger does not double-rollover.

## Rolling over many budgets at once (v0.1.25.29)

When a tenant has many budgets (workspace, app, workflow ladders), rolling each one over with individual `POST /v1/admin/budgets/fund` calls is fine but tedious — and leaves a tiny drift window between calls. Budget bulk-action lets you do the whole tenant atomically in one request.

::: warning `filter.tenant_id` is REQUIRED — one bulk call per tenant
Cross-tenant budget bulk is explicitly out of scope. The server returns `400 INVALID_REQUEST` if `filter.tenant_id` is missing or blank. **If you're rolling over a fleet, loop once per tenant** — do not attempt a single bulk call across multiple tenants. This is enforced to cap blast radius: a bad filter wipes at most one tenant's spend, not your whole fleet.
:::

```bash
curl -X POST http://localhost:7979/v1/admin/budgets/bulk-action \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "filter": { "tenant_id": "acme-corp", "unit": "USD_MICROCENTS" },
    "action": "RESET_SPENT",
    "amount": { "amount": 1000000, "unit": "USD_MICROCENTS" },
    "expected_count": 8,
    "idempotency_key": "period-rollover-2026-05-01-acme"
  }'
```

### Fleet-scale rollover pattern

```bash
# Per-tenant loop — substitute your tenant source of truth
for TENANT in $(cat tenants-to-roll.txt); do
  COUNT=$(curl -s "http://localhost:7979/v1/admin/budgets?tenant_id=$TENANT&unit=USD_MICROCENTS" \
    -H "X-Admin-API-Key: $ADMIN_KEY" | jq '.ledgers | length')

  curl -X POST http://localhost:7979/v1/admin/budgets/bulk-action \
    -H "X-Admin-API-Key: $ADMIN_KEY" \
    -H "Content-Type: application/json" \
    -d "{
      \"filter\": { \"tenant_id\": \"$TENANT\", \"unit\": \"USD_MICROCENTS\" },
      \"action\": \"RESET_SPENT\",
      \"amount\": { \"amount\": 1000000, \"unit\": \"USD_MICROCENTS\" },
      \"expected_count\": $COUNT,
      \"idempotency_key\": \"period-rollover-2026-05-01-$TENANT\"
    }"
done
```

The idempotency key **must** encode both the period and the tenant (`period-rollover-{date}-{tenant_id}`). A static key like `period-rollover-acme` replays the first month's response on month two — the bulk-action returns `200 OK` but mutates nothing. Including the date makes the key unique per billing boundary; including the tenant makes it unique per iteration.

### `amount` vs. `spent`

- **`amount`** (REQUIRED) is the new `allocated` ceiling for every matched budget after the reset. One value applies to all rows.
- **`spent`** (optional, `RESET_SPENT`-only) overrides the default-zero `spent` value after the reset. Use it for prorated signups mid-period or migrations that import an existing consumption number.

If each budget needs a different `allocated`, `amount` is too coarse — fall back to per-budget `POST /v1/admin/budgets/fund` calls.

### Prerequisites and per-row outcomes

- **Budgets must be ACTIVE.** `RESET_SPENT` against a `FROZEN` or `CLOSED` budget lands in the `failed[]` bucket with `error_code=INVALID_TRANSITION`. If you intentionally froze some budgets during a dispute, either unfreeze them first or add `status=ACTIVE` to the filter to skip them: `"filter": { "tenant_id": "acme-corp", "unit": "USD_MICROCENTS", "status": "ACTIVE" }`.
- **Other per-row outcomes:** `BUDGET_EXCEEDED` (only matters for DEBIT, not RESET_SPENT), `NOT_FOUND` (ledger deleted mid-operation), `PERMISSION_DENIED`, `INTERNAL_ERROR`.
- The 500-row cap and 15-minute idempotency window apply the same as for tenants/webhooks bulk-action.
- Emits one `budget.reset_spent` event per matched budget.
- Emits one audit row for the whole invocation with per-budget `succeeded_ids` / `failed_rows` — triageable from the audit log alone (v0.1.25.30+).

See [Using bulk actions for tenants, webhooks, and budgets](/how-to/using-bulk-actions-for-tenants-and-webhooks#budget-bulk-action) for the full envelope, safety gates, and per-row error codes.

## Common mistakes

- **Using `RESET` when you meant `RESET_SPENT`.** `RESET` rewrites `allocated` — it does not clear spend. If you call `RESET` with the same `amount` as the previous period, you've changed nothing. Use `RESET_SPENT` to zero out spend.
- **Zeroing out before in-flight reservations commit.** `RESET_SPENT` preserves reservations by design, so this is handled — but if you write custom tooling that manually sets `spent` to zero, remember to leave `reserved` alone.
- **Forgetting to roll over debt deliberately.** If your plan says debt should not carry between periods, issue an explicit `REPAY_DEBT` before the rollover (with the corresponding accounting entry in your billing system). `RESET_SPENT` on its own will leave debt untouched.
- **Not generating a unique idempotency key per period.** Reusing `rollover-acme` month after month means the second month is a replay of the first, returning the first month's response and moving nothing.

## Next steps

- [Budget Allocation and Management](/how-to/budget-allocation-and-management-in-cycles) — the full funding operation catalog
- [Admin API reference](/admin-api/) — OpenAPI definitions for `/v1/admin/budgets/fund`
- [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol) — `budget.reset_spent` event details
- [Multi-Tenant SaaS with Cycles](/how-to/multi-tenant-saas-with-cycles) — where rollover fits in a SaaS billing cycle



# Running the Cycles MCP Server over Streamable HTTP

The Cycles MCP server supports two transports:

- **STDIO** (default) — the AI client launches the server as a subprocess via `npx`. One server per developer, per machine.
- **Streamable HTTP** — the server runs as a long-lived process and clients connect remotely. One server, many clients. This is the current MCP remote transport; the older standalone HTTP+SSE transport is not implemented.

This page covers the Streamable HTTP transport. For STDIO setup with each AI client, see the per-client quickstarts: [Claude Desktop](/quickstart/mcp-claude-desktop), [Claude Code](/quickstart/mcp-claude-code), [Cursor](/quickstart/mcp-cursor), [Windsurf](/quickstart/mcp-windsurf).

## When to use HTTP instead of STDIO

| Situation | Transport |
|---|---|
| Single developer, local machine, one Cycles server | **STDIO** — simpler, zero process management |
| Team-wide MCP gateway shared across N developers | **HTTP** — one place to update, central auth |
| Remote / cloud deploy where the MCP server lives next to `cycles-server` | **HTTP** — co-located deploy |
| Agent runs in CI/CD or a Kubernetes pod | **HTTP** — sidecar pattern |
| You want to put auth, rate limiting, or audit logging in front of MCP | **HTTP** — terminate at a reverse proxy |

If you are not in one of the HTTP rows above, use STDIO. STDIO is simpler and avoids needing to think about network exposure, auth, or process supervision.

## Start the server with HTTP transport

```bash
export HOST=127.0.0.1
export MCP_HTTP_AUTH_TOKEN=replace-with-a-long-random-token
npx @runcycles/mcp-server --transport http
```

The server starts on port `3000`, binds to loopback, and requires the bearer token on `/mcp`. Do not copy the bare `npx ... --transport http` command into a reachable environment: without `HOST`, the current server binds all interfaces, and without `MCP_HTTP_AUTH_TOKEN`, `/mcp` is unauthenticated.

It exposes:

| Endpoint | Method | Purpose |
|---|---|---|
| `/health` | GET | Liveness probe — returns `{"status": "ok", "version": "..."}` |
| `/mcp` | POST | MCP Streamable HTTP endpoint (preferred for new clients) |
| `/mcp` | GET | Streamable HTTP SSE stream (server-to-client notifications) |
| `/mcp` | DELETE | Requests Streamable HTTP transport termination. Authenticate and restrict it like the other `/mcp` methods. |

### Configuration

| Variable | Default | Purpose |
|---|---|---|
| `PORT` | `3000` | HTTP port |
| `HOST` | all interfaces | HTTP bind address. Set `127.0.0.1` for loopback-only access. |
| `MCP_HTTP_AUTH_TOKEN` | — | Optional shared bearer token. When set, every `GET`, `POST`, and `DELETE` request to `/mcp` must send `Authorization: Bearer <token>`. `/health` remains public. |
| `CYCLES_API_KEY` | *(required in real mode)* | Cycles API key the server uses to talk to `cycles-server`. **Note:** in HTTP mode, this is the gateway's own key, not per-user. |
| `CYCLES_BASE_URL` | *(required in real mode)* | URL of `cycles-server` (e.g. `http://cycles-server:7878` if co-deployed) |
| `CYCLES_MOCK` | — | `"true"` to skip the backend and return mock responses (useful for client-integration tests) |
| `CYCLES_ALLOW_MOCK_IN_PRODUCTION` | `false` | Must be `"true"` to run mock mode with `NODE_ENV=production`; mock mode disables enforcement. |
| `CYCLES_DEFAULT_TENANT`, `CYCLES_DEFAULT_WORKSPACE`, `CYCLES_DEFAULT_APP`, `CYCLES_DEFAULT_WORKFLOW`, `CYCLES_DEFAULT_AGENT`, `CYCLES_DEFAULT_TOOLSET` | — | Fill omitted standard subject fields for subject-bearing tools. Explicit fields win; custom dimensions are never defaulted. |

If `MCP_HTTP_AUTH_TOKEN` is blank or whitespace-only, startup fails. If no token is configured while the server binds beyond loopback, startup prints a prominent warning. Built-in bearer auth is useful for a gateway with one shared credential; use an identity-aware proxy or API gateway when you need separate users, token rotation, rate limiting, or per-user policy.

## Worked example: docker-compose

The Cycles MCP server has no first-party container image yet, so the cleanest path today is a tiny Dockerfile that pins a server version, then run that image alongside your existing Cycles server. The example below assumes you already have a `cycles-server` running and reachable at some URL — see [Self-Hosting the Server](/quickstart/self-hosting-the-cycles-server) if you don't.

```dockerfile
# Dockerfile
FROM node:22-alpine
WORKDIR /app
RUN npm install --omit=dev @runcycles/mcp-server@0.6.0
EXPOSE 3000
CMD ["npx", "@runcycles/mcp-server", "--transport", "http"]
```

```yaml
# docker-compose.yml
services:
  cycles-mcp:
    build: .
    # Local/dev only. Bind the published port to host loopback. In production,
    # drop `ports:` and use `expose: ["3000"]` behind your gateway/proxy.
    ports:
      - "127.0.0.1:3000:3000"
    environment:
      CYCLES_API_KEY: ${CYCLES_API_KEY}
      CYCLES_BASE_URL: ${CYCLES_BASE_URL}
      MCP_HTTP_AUTH_TOKEN: ${MCP_HTTP_AUTH_TOKEN}
      PORT: "3000"
```

Run it:

```bash
export CYCLES_API_KEY=cyc_live_...
export CYCLES_BASE_URL=http://host.docker.internal:7878   # or wherever your Cycles server is
export MCP_HTTP_AUTH_TOKEN=replace-with-a-long-random-token
docker compose up -d --build
curl http://localhost:3000/health
# => {"status":"ok","version":"..."}
```

You can now point any HTTP-capable MCP client at `http://localhost:3000/mcp` and configure it to send the same bearer header. The Dockerfile pins the version for reproducibility; review and update that pin intentionally when upgrading. Keep the built-in token or put the service behind a reverse proxy/API gateway; use the latter when you need per-user identity or stronger network controls.

## Verify with MCP Inspector

Before debugging client-side wiring, prove the server itself works using the MCP reference client:

```bash
npx @modelcontextprotocol/inspector
```

In the Inspector UI, select **Streamable HTTP** as the transport and enter:

```
http://localhost:3000/mcp
```

If `MCP_HTTP_AUTH_TOKEN` is configured, add an `Authorization` header with the value `Bearer <token>` in the Inspector connection settings.

List tools (you should see `cycles_reserve`, `cycles_commit`, `cycles_check_balance`, etc.) and call `cycles_check_balance` with a tenant you know exists. If that works, any subsequent connection failures are client-config issues, not server issues.

## Connecting an MCP client to a remote server

### Claude Code

Claude Code has first-party CLI support for remote HTTP MCP servers:

```bash
claude mcp add --transport http cycles https://mcp.example.com/mcp
```

For local testing against the docker-compose above:

```bash
claude mcp add --transport http \
  --header "Authorization: Bearer $MCP_HTTP_AUTH_TOKEN" \
  cycles http://localhost:3000/mcp
```

The local command uses the built-in bearer token configured in the docker-compose example. For another remote server, send the authentication headers required by that server; omit `--header` only when the endpoint is intentionally unauthenticated. Claude Code stores this local-scope server configuration outside the project, so the expanded token is not committed to the repository.

### Other clients (config shape varies)

For clients that take JSON config rather than a CLI, the shape replaces the STDIO `command`/`args` launch with a remote URL. The exact keys differ across clients and are still evolving — some use just `"url"`, others require an explicit `"type": "http"` discriminator. Two examples seen in the wild:

```json
{
  "mcpServers": {
    "cycles": {
      "url": "https://mcp.example.com/mcp"
    }
  }
}
```

```json
{
  "mcpServers": {
    "cycles": {
      "type": "http",
      "url": "https://mcp.example.com/mcp"
    }
  }
}
```

Windsurf documents stdio, HTTP, and SSE transports. Claude Code supports remote HTTP via the CLI above. Other clients may vary by release channel — check the client docs before assuming a JSON shape. STDIO is widely supported and is the right fallback for the local clients covered by these guides while remote support stabilizes.

## Auth, scope derivation, and security

- **The MCP server's `CYCLES_API_KEY` is the gateway's identity, not the end user's.** Every request to Cycles authenticates as that one key. `MCP_HTTP_AUTH_TOKEN` protects the MCP endpoint with one shared credential; it does not create per-user Cycles identities.
- **End-user attribution is not injected automatically.** The MCP schemas do not accept a reservation `actor` field. A client or identity-aware tool harness can attach audit context through `action.tags` or `metadata` on subject-bearing operations; `metrics.custom` is available only on commit and create-event calls. These fields add observability but do not determine the budget scope. If you need per-user or per-tenant enforcement, map the authenticated identity to an explicit subject policy before the Cycles call, or use separate gateway/API-key identities per boundary. See [Custom Field Resolvers](/how-to/custom-field-resolvers-in-cycles).
- **Scope derivation behaves identically over HTTP.** `cycles_reserve`, `cycles_decide`, and `cycles_create_event` accept the subject hierarchy ([tenant → workspace → app → workflow → agent → toolset](/concepts/exposure-why-rate-limits-leave-agents-unbounded)); `cycles_check_balance` accepts the corresponding filters. Commit, release, and extend operate on an existing `reservationId` and do not accept a new subject.
- **Protect every reachable `/mcp` endpoint.** Use `MCP_HTTP_AUTH_TOKEN` for shared-token access, or put the service behind nginx/caddy/Traefik, mTLS, an API gateway, or a private network. Prefer an identity-aware gateway when users need distinct credentials or policies.
- **Validate browser origins at the proxy.** MCP's Streamable HTTP specification requires servers to validate the `Origin` header to prevent DNS rebinding. Cycles MCP Server v0.6.0 does not yet perform that validation itself. Until it does, reject unexpected `Origin` values at the reverse proxy or API gateway; keep loopback deployments authenticated as defense in depth.
- **Health check is intentionally unauthenticated.** `/health` returns version information for load balancers. Built-in bearer auth, when configured, applies to every `/mcp` method.

## Known limitations

- **No built-in per-user auth.** The built-in bearer token is shared. If the goal is per-developer attribution, use an identity-aware gateway or STDIO with a separate Cycles API key per developer.
- **No application-level `Origin` validation in v0.6.0.** Put HTTP deployments behind a proxy that validates browser origins, and do not expose an unauthenticated listener even when it binds to loopback.
- **No first-party container image.** A pinned GHCR image will land once HTTP demand is validated. Until then, the version-pinned Dockerfile above is the recommended pattern.
- **The server is stateless.** It issues no session IDs, so any replica can serve any request — no sticky sessions needed. Restarts are safe from a transport perspective; retry an interrupted mutating tool call with the same caller-supplied `idempotencyKey`. A new key represents a new operation and can create a duplicate hold, charge, extension, event, or evidence artifact.

## Next steps

- [Integrating Cycles with MCP](/how-to/integrating-cycles-with-mcp) — advanced patterns: preflight, degradation, long-running ops, fire-and-forget events
- [Per-client STDIO quickstarts](/quickstart/getting-started-with-the-mcp-server) — when STDIO is the right call
- [API Key Management](/how-to/api-key-management-in-cycles) — rotation and lifecycle for the gateway's key
- [Multi-Tenant Operations](/guides/multi-tenant-operations) — how scope hierarchy works end-to-end



# Searching and Sorting Admin List Endpoints

The admin and runtime planes expose the list endpoints below. The admin list endpoints share a consistent query-parameter vocabulary for filtering, searching, sorting, and paginating; `/v1/reservations` uses the same sort/cursor conventions for runtime reservations. This page is the practical reference for using them from curl, scripts, and operator tools.

The endpoints:

| Endpoint | Plane | Added / enhanced |
|----------|-------|------------------|
| `GET /v1/admin/tenants` | Admin | sort v0.1.25.24, search v0.1.25.25 |
| `GET /v1/admin/budgets` | Admin | filters v0.1.25.22, sort v0.1.25.24, search v0.1.25.25 |
| `GET /v1/admin/api-keys` | Admin | cross-tenant v0.1.25.22, sort v0.1.25.24, search v0.1.25.25 |
| `GET /v1/admin/webhooks` | Admin | sort v0.1.25.24, search v0.1.25.25 |
| `GET /v1/admin/events` | Admin | sort v0.1.25.24, search v0.1.25.25 |
| `GET /v1/admin/audit/logs` | Admin | failure capture v0.1.25.20, sort v0.1.25.24, search v0.1.25.25 |
| `GET /v1/reservations` | Runtime | sort v0.1.25.12 |

The parameters were added compatibly: older servers that predate a parameter may ignore it rather than erroring. Current servers still validate the parameters they recognize, so unsupported `sort_by`, invalid `sort_dir`, out-of-range `limit`, and `search` values over 128 characters return `400 INVALID_REQUEST`.

## Parameter vocabulary

### `search` (v0.1.25.25+)

A case-insensitive substring match over the endpoint's searchable identifier fields. Maximum 128 characters. Longer strings return `400 INVALID_REQUEST`.

| Endpoint | Fields matched by `search` |
|----------|---------------------------|
| `/v1/admin/tenants` | `tenant_id`, `name` |
| `/v1/admin/budgets` | `tenant_id`, `scope` |
| `/v1/admin/api-keys` | `key_id`, `name` |
| `/v1/admin/webhooks` | `subscription_id`, `url` |
| `/v1/admin/events` | `correlation_id`, `scope` |
| `/v1/admin/audit/logs` | `resource_id`, `log_id`, `error_code`, `operation` |

`search` is applied after other filters (`status`, `parent_tenant_id`, etc.) and is combined with them using AND semantics.

### `sort_by` and `sort_dir`

`sort_by` names the field to order on. `sort_dir` is `asc` or `desc`; when omitted it defaults to `desc`.

| Endpoint | Supported `sort_by` values |
|----------|---------------------------|
| `/v1/admin/tenants` | `tenant_id`, `name`, `status`, `created_at` |
| `/v1/admin/budgets` | `tenant_id`, `scope`, `unit`, `status`, `commit_overage_policy`, `utilization`, `debt` |
| `/v1/admin/api-keys` | `key_id`, `name`, `tenant_id`, `status`, `created_at`, `expires_at` |
| `/v1/admin/webhooks` | `url`, `tenant_id`, `status`, `consecutive_failures` |
| `/v1/admin/events` | `event_type`, `category`, `scope`, `tenant_id`, `timestamp` |
| `/v1/admin/audit/logs` | `timestamp`, `operation`, `resource_type`, `tenant_id`, `key_id`, `status` |
| `/v1/reservations` | `reservation_id`, `tenant`, `scope_path`, `status`, `reserved`, `created_at_ms`, `expires_at_ms` |

Unknown `sort_by` or `sort_dir` values return `400 INVALID_REQUEST`. The reservation endpoint sorts the integer `amount` within the `reserved` key (well-defined under v0's single-unit-per-reservation invariant); `scope_path` sorts the canonical scope string lexicographically.

::: warning Default order
Current admin-list defaults are endpoint-specific: tenants and API keys use `created_at desc`; budgets use `utilization desc`; webhooks use `consecutive_failures desc`; events and audit logs use `timestamp desc`. If a script relies on row order, pass `sort_by` and `sort_dir` explicitly. `/v1/reservations` retains its legacy default order unless `sort_by` is provided.
:::

### `cursor`, `limit`, `has_more`, `next_cursor`

Pagination is cursor-based:

- `limit` — maximum results per page. The default is 50 on every endpoint. The runtime `/v1/reservations` cap is 200. The capped admin endpoints use 100 (`/v1/admin/tenants`, `/v1/admin/webhooks`, `/v1/admin/events`); `/v1/admin/budgets`, `/v1/admin/api-keys`, and `/v1/admin/audit/logs` declare no maximum. Out-of-range values on capped endpoints return `400 INVALID_REQUEST`.
- `cursor` — opaque string from a previous response's `next_cursor`. Do not construct or modify it.
- `has_more` — boolean in the response. `true` means there is at least one more page.
- `next_cursor` — the value to pass as `cursor` on the next call. Absent when `has_more` is `false`.

### Cursor binding

When `sort_by` is provided, the returned cursor encodes the sort key so "Load more" continues in sort order. Current admin servers bind the cursor to the result-set parameters: reusing it after changing the sort key, direction, or filters returns `400 INVALID_REQUEST`. The response uses the generic error code rather than a cursor-specific code.

**Reset the cursor whenever you change the sort key, sort direction, or any filter.** The client's job is to either preserve those parameters across all pages of a traversal or start over from page one.

### Cross-tenant listing (admin only)

Omitting the `tenant_id` query parameter on `/v1/admin/api-keys`, `/v1/admin/webhooks`, `/v1/admin/budgets`, `/v1/admin/events`, and `/v1/admin/audit/logs` returns rows across all tenants. Authentication must be via `X-Admin-API-Key` for cross-tenant access — tenant-scoped `X-Cycles-API-Key` calls are limited to their own tenant.

As an implementation note, the reference server's cross-tenant API-key and budget walks use composite cursors (conceptually `(tenant_id, key_id)` / `(tenant_id, ledger_id)`) so traversal remains stable across tenants — but cursor shape is not part of the spec. Treat every `next_cursor` as opaque regardless of endpoint.

## Forward-compatible preview filters

Two admin list surfaces accept v0.1.26-preview filters even on v0.1.25.x reference admin servers:

| Endpoint | Preview filters | v0.1.25.x behavior |
|---|---|---|
| `GET /v1/admin/tenants` | `observe_mode=DISABLED\|OBSERVE\|ENFORCE` | Accepted for compatibility; not applied until observe mode is implemented |
| `GET /v1/admin/policies` | `has_action_quotas`, `references_action_kind` | Accepted for compatibility; not applied until action-governance policy fields are implemented |

`GET /v1/admin/policies` is cursor-paginated but is not one of the six search/sort endpoints listed above. Its current filters are `tenant_id`, `scope_pattern`, `status`, `cursor`, and `limit`; the preview filters are documented in [Action Governance Preview](/protocol/action-governance-preview-in-cycles).

## Recipes

### Oldest-expiring active reservations

Incident response — find reservations about to expire that are holding budget:

```bash
curl -G "http://localhost:7878/v1/reservations" \
  -H "X-Cycles-API-Key: $TENANT_API_KEY" \
  --data-urlencode "status=ACTIVE" \
  --data-urlencode "sort_by=expires_at_ms" \
  --data-urlencode "sort_dir=asc" \
  --data-urlencode "limit=50" | jq .
```

### Most-utilized budgets

Capacity review — find the budgets closest to exhaustion:

```bash
curl -G "http://localhost:7979/v1/admin/budgets" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  --data-urlencode "sort_by=utilization" \
  --data-urlencode "sort_dir=desc" \
  --data-urlencode "limit=25" | jq .
```

### Over-limit budgets with debt

Debt review — find scopes currently in overdraft:

```bash
curl -G "http://localhost:7979/v1/admin/budgets" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  --data-urlencode "over_limit=true" \
  --data-urlencode "has_debt=true" \
  --data-urlencode "sort_by=debt" \
  --data-urlencode "sort_dir=desc" | jq .
```

`over_limit`, `has_debt`, and `utilization_min` / `utilization_max` are budget-specific filters added in v0.1.25.22.

### Webhooks about to auto-disable

Health check — find subscriptions approaching the `disable_after_failures` threshold:

```bash
curl -G "http://localhost:7979/v1/admin/webhooks" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  --data-urlencode "sort_by=consecutive_failures" \
  --data-urlencode "sort_dir=desc" \
  --data-urlencode "limit=10" | jq .
```

### Search across tenants for a key

Audit — find every API key whose name contains "integration":

```bash
curl -G "http://localhost:7979/v1/admin/api-keys" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  --data-urlencode "search=integration" \
  --data-urlencode "sort_by=created_at" \
  --data-urlencode "sort_dir=desc" | jq .
```

## Audit log filter DSL (v0.1.25.27)

`GET /v1/admin/audit/logs` supports a richer filter DSL than the other list endpoints. In addition to `search`, `sort_by`, and `sort_dir`, it accepts:

| Parameter | Type | Purpose |
|---|---|---|
| `error_code` | array (max 25) | Exact-or-IN-list on `error_code`. Comma-separated form (`?error_code=a,b`). NULL (success rows) does not match. |
| `error_code_exclude` | array (max 25) | NOT-IN-list. NULL always passes. Combine with `error_code` via AND. |
| `status_min` | integer 100–599 | Inclusive lower bound. Mutually exclusive with exact `status`. |
| `status_max` | integer 100–599 | Inclusive upper bound. `status_min > status_max` returns 400. |
| `operation` | array (max 25) | Promoted from scalar. `?operation=createBudget,updateBudget`. |
| `resource_type` | array (max 25) | Same shape. |
| `trace_id` | 32-hex | Exact-match JOIN across events and webhook deliveries (v0.1.25.31). See [Correlation and Tracing](/protocol/correlation-and-tracing-in-cycles). |
| `request_id` | string | Exact-match on per-HTTP-request id (v0.1.25.31). |

Also, `search` on `listAuditLogs` was extended to match `error_code` and `operation` in addition to `resource_id` / `log_id` — useful when you remember "`BUDGET_EXCEEDED` was involved" but not the full resource id.

Audit metadata is returned on each row but is not exposed as `metadata.*` query parameters. For metadata-only facts such as `metadata.actor_type` or bulk-action `metadata.idempotency_key`, narrow with top-level filters (`operation`, `tenant_id`, `resource_type`, `resource_id`, `trace_id`, `request_id`, or time range), then inspect the expanded row or exported JSON.

```bash
# 5xx failures on budget endpoints in the last hour, not counting known idempotency noise
curl -G 'http://localhost:7979/v1/admin/audit/logs' \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  --data-urlencode "status_min=500" \
  --data-urlencode "resource_type=budget" \
  --data-urlencode "error_code_exclude=IDEMPOTENCY_MISMATCH" \
  --data-urlencode "from=2026-04-18T12:00:00Z" | jq .

# Everything admins did cross-tenant today
curl -G 'http://localhost:7979/v1/admin/audit/logs' \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  --data-urlencode "tenant_id=__admin__" \
  --data-urlencode "from=2026-04-18T00:00:00Z" | jq .
```

### Tenant sentinels (v0.1.25.28)

- `__admin__` — admin-plane operations not scoped to a tenant (governance ops, cross-tenant reads, admin-plane 4xx/5xx). Authenticated-tier retention.
- `__unauth__` — pre-authentication failures. Unauthenticated-tier retention, subject to `audit.sample.unauthenticated`.

v0.1.25.28 renamed the previous single `<unauthenticated>` sentinel. Historical rows keep their `<unauthenticated>` literal and age out under the unauth-tier TTL; migrate queries to `__unauth__` (pre-auth failures only) or `__admin__` (new platform-admin slice).

## Hydration warning on sorted reservation listings

On `/v1/reservations`, `sort_by=created_at_ms` can use the optional, completeness-gated created-at sorted index in runtime v0.1.25.54 and later. The index is used only when it is enabled and ready; otherwise the server falls back without returning incomplete results. Track `cycles_reservations_created_at_index_reads_total{outcome=...}` to distinguish index reads from disabled, not-ready, drift, and error fallbacks.

The other six reservation sort keys—and `created_at_ms` when the index cannot be used—hydrate all matches, then sort and slice. If a sorted query hydrates 2,000 or more rows, the server logs a WARN so operators can narrow filters. Rows beyond 2,000 are not truncated in v0.1.25.39+.

For faster, more predictable queries, narrow the filter: add `status`, `idempotency_key`, a time window, or a subject field (`workspace`, `app`, `workflow`, `agent`, `toolset`). The admin list endpoints keep their own endpoint-specific sort behavior.

## Error reference

| `error` | Meaning |
|---------|---------|
| `INVALID_REQUEST` | Unknown `sort_by`, unknown `sort_dir`, out-of-range `limit`, `search` over 128 chars, or a cursor reused with different result-set parameters |
| `FORBIDDEN` | Tenant-scoped key attempted a cross-tenant listing |
| `UNAUTHORIZED` | Invalid API key |

The error code is carried in the `error` field of the standard `ErrorResponse` body. There is no cursor-specific error code; reset pagination after any result-set parameter changes.

## Next steps

- [Admin API reference](/admin-api/) — full OpenAPI for each endpoint
- [Reservation Recovery and Listing](/protocol/reservation-recovery-and-listing-in-cycles) — reservation-specific sort and recovery patterns
- [Using Bulk Actions](/how-to/using-bulk-actions-for-tenants-and-webhooks) — bulk actions take the same filter shape as the list endpoints
- [API Key Management](/how-to/api-key-management-in-cycles) — cross-tenant key listing in practice



# Security Hardening

This guide covers security best practices for a production Cycles deployment.

::: warning Critical
The Admin Server (port 7979) should **never** be exposed to the public internet. It has full control over tenants, API keys, and budgets.
:::

## Network isolation

### Separate management and runtime planes

All Cycles services — Server, Admin Server, Events Service — run on the internal network. Only a load balancer should be exposed to application traffic. The Admin Server (port 7979), Events Service app/management ports (7980/9980), Cycles Server (port 7878), and Redis (port 6379) should **never be accessible from the public internet**.

<NetworkZones />

### Firewall rules

| Source | Destination | Port | Allow |
|---|---|---|---|
| Public internet | Load Balancer | 443 (HTTPS) | Yes |
| Load Balancer | Cycles Server | 7878 (internal) | Yes |
| Application servers (internal) | Cycles Server | 7878 | Yes |
| Operations team (VPN) | Admin Server | 7979 | Yes |
| Cycles Server | Redis | 6379 | Yes |
| Admin Server | Redis | 6379 | Yes |
| Events Service | Redis | 6379 | Yes |
| Events Service | External webhook endpoints | 443 (HTTPS) | Yes |
| Public internet | Cycles Server | 7878 | **No** |
| Public internet | Admin Server | 7979 | **No** |
| Public internet | Events Service | 7980/9980 | **No** |
| Public internet | Redis | 6379 | **No** |

## Redis security

### Authentication

Always set a strong Redis password in production:

```yaml
# docker-compose.yml
redis:
  image: redis:7-alpine
  command: redis-server --requirepass ${REDIS_PASSWORD} --appendonly yes

cycles-server:
  environment:
    REDIS_PASSWORD: ${REDIS_PASSWORD}

cycles-admin:
  environment:
    REDIS_PASSWORD: ${REDIS_PASSWORD}

cycles-events:
  environment:
    REDIS_PASSWORD: ${REDIS_PASSWORD}
```

All three services — Server, Admin Server, and Events Service — connect to the same Redis, so all three need `REDIS_PASSWORD` set.

Generate a strong password:

```bash
openssl rand -base64 32
```

### Redis TLS

For environments where Redis traffic crosses network boundaries, enable TLS:

```conf
# redis.conf
tls-port 6380
port 0  # Disable non-TLS port
tls-cert-file /etc/redis/tls/redis.crt
tls-key-file /etc/redis/tls/redis.key
tls-ca-cert-file /etc/redis/tls/ca.crt
```

### Redis ACLs

Restrict the Cycles service account to the key patterns the services actually use. All three services (Server, Admin Server, Events Service) share this account, so the pattern list must cover every prefix — omitting the event/delivery/dispatch/evidence prefixes breaks the Events Service:

```conf
# redis.conf
user cycles on >${REDIS_PASSWORD} ~tenant:* ~budget:* ~budgets:* ~reservation:* ~reserve:* ~idem:* ~idempotency:* ~apikey:* ~apikeys:* ~policy:* ~policies:* ~audit:* ~event:* ~events:* ~delivery:* ~deliveries:* ~dispatch:* ~webhook:* ~webhooks:* ~evidence:* ~config:* ~replay:* +@all
user default off
```

Key prefixes by service:

| Service | Key prefixes |
|---|---|
| Cycles Server | `tenant:`, `budget:`, `reservation:`, `reserve:`, `idem:`, `apikey:`, `audit:`, `event:`, `events:`, `delivery:`, `deliveries:`, `dispatch:`, `webhook:`, `webhooks:`, `evidence:` |
| Admin Server | `tenant:`, `budget:`, `budgets:`, `apikey:`, `apikeys:`, `policy:`, `policies:`, `audit:`, `event:`, `events:`, `delivery:`, `deliveries:`, `dispatch:`, `webhook:`, `webhooks:`, `replay:`, `idem:`, `config:` |
| Events Service | `event:`, `events:`, `delivery:`, `deliveries:`, `dispatch:`, `webhook:`, `evidence:`, `config:` |

### Disable dangerous commands

```conf
rename-command FLUSHDB ""
rename-command FLUSHALL ""
rename-command DEBUG ""
rename-command CONFIG ""
```

## API key management

### Key rotation

API keys should be rotated regularly:

1. Create a new key with the same permissions
2. Update the application configuration to use the new key
3. Verify the application works with the new key
4. Revoke the old key

```bash
# 1. Create new key
NEW_KEY=$(curl -s -X POST http://localhost:7979/v1/admin/api-keys \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -d '{
    "tenant_id": "acme-corp",
    "name": "prod-key-v2",
    "permissions": ["reservations:create","reservations:commit","reservations:release","reservations:extend","balances:read"]
  }' | jq -r '.key_secret')

# 2. Update application config (deploy with new key)
# 3. Verify application health

# 4. Revoke old key
curl -s -X DELETE "http://localhost:7979/v1/admin/api-keys/${OLD_KEY_ID}" \
  -H "X-Admin-API-Key: $ADMIN_KEY"
```

### Least-privilege permissions

Only grant the permissions each component needs:

| Component | Permissions needed |
|---|---|
| Application (runtime) | `reservations:create`, `reservations:commit`, `reservations:release`, `reservations:extend`, `balances:read` |
| Monitoring service | `balances:read`, `reservations:list` |
| Batch processor | `reservations:create`, `reservations:commit` |

Don't give application keys full permissions when they only need a subset.

### Admin key security

The `ADMIN_API_KEY` (used in the `X-Admin-API-Key` header) has full administrative access. Protect it:

- Store in a secrets manager (AWS Secrets Manager, HashiCorp Vault, etc.)
- Never commit to source control
- Rotate on a schedule
- Limit who can access it

### Key storage in applications

```bash
# Good: environment variables from secrets manager
export CYCLES_API_KEY=$(aws secretsmanager get-secret-value --secret-id cycles/api-key --query SecretString --output text)

# Bad: hardcoded in source code
# CYCLES_API_KEY = "cyc_live_abc123..."  # NEVER DO THIS
```

## Audit logging

The Admin Server records audit logs for administrative operations. Use these for:

- **Compliance:** Track who created/modified/revoked API keys
- **Incident response:** Determine when a tenant or budget was changed
- **Access review:** Identify unused or over-privileged keys

Query audit logs:

```bash
curl -s "http://localhost:7979/v1/admin/audit/logs?tenant_id=acme-corp&limit=50" \
  -H "X-Admin-API-Key: $ADMIN_KEY" | jq .
```

### Retention policy

- **Hot storage (Redis):** 400 days for authenticated admin operations, 30 days for unauthenticated (failed-auth) entries — queryable via the API. Configurable via `AUDIT_RETENTION_AUTHENTICATED_DAYS` and `AUDIT_RETENTION_UNAUTHENTICATED_DAYS`.
- **Cold storage:** Export to S3/GCS/etc. for long-term retention (1+ year recommended for compliance)

Don't confuse audit retention with event retention: 90 days is the event TTL (`EVENT_TTL_DAYS`), not the audit log retention.

Set up a periodic export job to archive audit logs before they expire from Redis.

## Actuator authentication

Since Cycles Server `0.1.25.45`, operational endpoints — `/actuator/prometheus`, `/actuator/info`, the aggregate `/actuator/health`, API docs, and Swagger — require the `X-Admin-API-Key` header. Only the orchestrator probes `/actuator/health/liveness` and `/actuator/health/readiness` remain public. The Admin Server enforces the same rule. Prometheus scrapers that read these endpoints must send the header.

The Events Service exposes its actuator on a separate management port (9980), which is unauthenticated by design — the separate port is its isolation mechanism. Never publish port 9980 to the host or expose it beyond the internal network; isolate it at the network layer and scrape Prometheus from inside that boundary.

## TLS for client-to-server communication

Terminate TLS at the load balancer or reverse proxy. See the [Production Operations Guide](/how-to/production-operations-guide) for nginx configuration.

For service-to-service communication within a trusted network (e.g., Kubernetes cluster), plain HTTP to the Cycles Server is acceptable if network policies restrict access.

## Container security

### Run as non-root

The Cycles Server Docker images run as a non-root user by default. Verify (the image's entrypoint launches the Java server, so `whoami` must be passed with `--entrypoint` — appending it as a command argument is ignored):

```bash
docker run --rm --entrypoint whoami ghcr.io/runcycles/cycles-server:0.1.25.59
```

### Pin image versions

Use specific version tags, not `latest`:

```yaml
image: ghcr.io/runcycles/cycles-server:0.1.25.59  # Pinned
# NOT: ghcr.io/runcycles/cycles-server:latest   # Unpinned
```

### Read-only filesystem

Mount the container filesystem as read-only:

```yaml
cycles-server:
  image: ghcr.io/runcycles/cycles-server:0.1.25.59
  read_only: true
  tmpfs:
    - /tmp
```

## Security checklist

- [ ] Admin Server not accessible from public internet
- [ ] Redis not accessible from public internet
- [ ] Redis password set and stored in secrets manager
- [ ] API keys use least-privilege permissions
- [ ] Admin key stored in secrets manager, not in source control
- [ ] TLS termination configured for client-facing traffic
- [ ] Container images pinned to specific versions
- [ ] Audit log retention policy defined
- [ ] Key rotation schedule established
- [ ] Dangerous Redis commands disabled
- [ ] Actuator/management endpoints protected: `X-Admin-API-Key` required, Events Service port 9980 not published
- [ ] `allow_http=false` in webhook security config (all environments)
- [ ] `WEBHOOK_SECRET_ENCRYPTION_KEY` generated and stored in secrets manager
- [ ] Webhook URL security: HTTPS enforced, private CIDR ranges blocked
- [ ] Signing secret rotation procedure documented

## Webhook Security

### Signing secret encryption at rest

Webhook signing secrets are encrypted in Redis using AES-256-GCM. All three services must share the same key via `WEBHOOK_SECRET_ENCRYPTION_KEY`. Generate with `openssl rand -base64 32`. Store in a secrets manager — not in source code.

### SSRF prevention

Webhook URLs resolving to private IPs are blocked by default (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, loopback, link-local). HTTP is rejected by default — keep `allow_http=false` **always**, in every environment, not just production. Configure via `PUT /v1/admin/config/webhook-security`.

### Signing secret rotation

`PATCH /v1/admin/webhooks/{id}` with a new `signing_secret`. Update the consumer to verify the new secret. In-flight retries will use the old secret until retried with the new one.

### Encryption key rotation

Rotating `WEBHOOK_SECRET_ENCRYPTION_KEY` requires decrypting all secrets with the old key, re-encrypting with the new key, and restarting all services simultaneously.

## Next steps

- [Security Overview](/security) — data residency, audit trail, compliance posture
- [Production Operations Guide](/how-to/production-operations-guide) — deployment and infrastructure
- [Monitoring and Alerting](/how-to/monitoring-and-alerting) — metrics and alerting
- [API Key Management](/how-to/api-key-management-in-cycles) — key lifecycle management



# Shadow Mode in Cycles: How to Roll Out Budget Enforcement Without Breaking Production

Most teams do not struggle with the idea of budget enforcement.

They struggle with the rollout.

They know autonomous systems can loop, retry, fan out across tools, and create unbounded cost or side effects. They know rate limits and dashboards are not enough.

But they also know that hard enforcement can break production if the policy is wrong.

That is why shadow mode matters.

Shadow mode lets a team evaluate Cycles policies against real traffic **without yet blocking execution**.

It is how you move from theory to production safely.

## Why shadow mode is necessary

In practice, most teams do not know the correct budget thresholds on day one.

They may have good instincts, but they usually do not yet know:

- how much a typical workflow actually consumes
- how often retries happen
- which workflows are naturally bursty
- which tenants have unusual usage patterns
- how often expensive tool paths are taken
- how much estimated usage differs from actual usage

If you enforce too early, you risk false denials, broken workflows, frustrated users, and emergency rollback.

If you never enforce at all, you stay stuck in observability mode forever.

Shadow mode is the bridge between those two states.

## What shadow mode is

In the Cycles protocol, shadow mode is enabled by setting `dry_run: true` on a reservation request. The server evaluates the same reservation and budget logic it would use in enforcement mode, but instead of blocking execution, it returns what **would have happened**—the decision, caps, and affected scopes—in the response. No balances are modified, no reservation is created, and no commit or release is required.

The current reference server emits `reservation.denied` for a denied dry-run or `decide` evaluation, and an enabled evidence pipeline may retain a signed evaluation artifact. It does not emit a corresponding allowed lifecycle event or know the eventual external outcome. Record every response and actual outcome in the application when you need a complete shadow dataset.

That means your system can answer questions like:

- would this action have been allowed?
- which scope would have denied it?
- how often would this execution exceed a workflow ledger mapped to its run ID?
- which tenants are consistently near their limits?
- how accurate are our estimates versus actual usage recorded by the application?

In other words, shadow mode gives you production-grade policy feedback without introducing production-grade disruption.

The protocol's v0.1.26 admin extension previews tenant-level `observe_mode` fields. The current v0.1.25.x server accepts those fields for forward compatibility but does not apply fleet-wide shadow semantics or emit observed decisions from them. Use per-request `dry_run: true` and application-side logging today.

Cycles' standard budget hierarchy is `tenant → workspace → app → workflow → agent → toolset`. If you want a separate ledger for each run, map the run ID to `subjects.workflow`; putting a run ID only in `dimensions` adds attribution but does not create a budget scope.

## What shadow mode is not

Shadow mode is not fake traffic.

It is not synthetic testing.

It is not a spreadsheet exercise.

It is not simply “log more metrics.”

It is real policy evaluation against real production behavior, with the difference that the outcome is observed instead of enforced.

That distinction matters.

Teams often think they can skip shadow mode by reading dashboards or replaying logs. Sometimes that helps, but it rarely captures the full reality of live autonomous execution, especially under retries, concurrency, and partial failure.

## The core idea

The Cycles control model is:

1. declare intent
2. reserve exposure
3. execute
4. commit actual usage or release the remainder

In shadow mode, the same model still runs conceptually, but reservation failures become **signals** rather than **hard stops**.

Instead of saying:

::: info
deny this action
:::

the system says:

::: info
this action would have been denied under the current policy
:::

That gives teams a safe way to tune policy before the consequences become user-facing.

## Why shadow mode matters for autonomous systems

Autonomous systems are harder to govern than simple request/response applications because behavior emerges over time.

The cost of a workflow is often not obvious from its starting point.

One execution may:

- call a model once and finish cheaply
- branch into multiple retrieval calls
- invoke several tools
- retry after partial failure
- recurse into additional steps
- continue running in the background

That means the correct budget boundaries are often learned empirically.

Shadow mode gives teams a way to learn those boundaries from reality instead of guessing.

## What teams should measure in shadow mode

A useful shadow rollout is not just “turn it on and watch logs.”

It should answer concrete questions.

### 1. Denial frequency

How often would actions have been denied?

This helps identify whether policy is too strict, too loose, or roughly calibrated.

### 2. Denial location

Which scope would have denied the action?

For example:

- tenant budget
- workflow budget
- workflow budget keyed by run ID

This shows whether the problem is broad account-level consumption or a local execution issue.

### 3. Estimate versus actual usage

How often are your reservations too high or too low?

If estimates are consistently inflated, you may create unnecessary policy pressure.  
If estimates are consistently too low, your controls may be less protective than expected.

Because a dry run creates no reservation to commit, record actual usage in application telemetry and join it to the logged dry-run response.

### 4. Workflow distribution

Which workflows consume the most exposure?  
Which ones are bursty?  
Which ones are stable?

This helps you decide where workflow-specific policies are worth adding.

### 5. Tenant distribution

Which tenants are close to limits?  
Which tenants have unusual patterns?  
Which ones would be most affected by enforcement?

This is especially important for multi-tenant platforms.

### 6. Runaway behavior indicators

Which runs show repeated retries, recursive tool usage, or unusually long chains of actions?

These are often the strongest signals that run-level limits need tuning.

## A practical rollout sequence

A safe shadow-mode rollout usually follows a progression.

### Phase 1: Instrument core actions

Start by evaluating the highest-value actions in shadow mode, such as:

- model calls
- expensive tool invocations
- side-effecting actions
- long-running workflow steps

Do not try to model every possible action on day one.

### Phase 2: Add the most important scopes

Start with a small set of budget scopes, usually:

- tenant
- workflow, optionally keyed by run ID

Those two often provide the clearest operational signal.

Workflow budgets can be added once you understand which process types deserve distinct treatment.

### Phase 3: Collect enough live behavior

Let the system observe enough real production traffic to expose variation.

The goal is not only to capture average behavior, but also:

- spikes
- retries
- partial failures
- unusual tenants
- edge-case workflows

### Phase 4: Review would-deny outcomes

Look at the actions that would have been denied.

Ask:

- would we actually want to stop this?
- should this degrade instead of deny?
- is the budget too strict?
- is the estimate too high?
- is a different scope the right boundary?

### Phase 5: Tune and repeat

Adjust policy, estimate strategy, or degradation logic.

Then continue observing until the system’s would-deny behavior matches operator intent closely enough to justify enforcement.

### Phase 6: Move selected paths to hard enforcement

You do not have to turn on enforcement everywhere at once.

A good rollout often begins with:

- the most expensive actions
- the most predictable workflows
- the most stable tenants
- the clearest runaway failure modes

That keeps the first production enforcement surface narrow and understandable.

## What good shadow-mode outcomes look like

A successful shadow period usually produces a few things.

### Clear budget boundaries

You begin to understand what reasonable tenant and workflow limits look like, including workflows keyed per execution when that mapping fits your application.

### Estimate quality improves

You learn whether your reservation estimates are directionally correct or need refinement.

### Denial logic becomes intentional

The team stops asking “what number should we pick?” and starts asking “what behavior do we want to allow, degrade, or stop?”

### Enforcement becomes safer

By the time hard enforcement begins, you have already seen the likely denial cases and adjusted policy accordingly.

That is the real value of shadow mode.

## Common mistakes in shadow rollouts

### Mistake 1: Treating shadow mode as a checkbox

Shadow mode is not useful unless someone reviews the results and tunes policy.

If nobody looks at the would-deny outcomes, shadow mode becomes passive logging.

### Mistake 2: Starting with too many scopes

If you begin with tenant, workspace, app, workflow, agent, and toolset scopes all at once, it becomes difficult to understand what is actually driving decisions.

Start small.

### Mistake 3: Using shadow mode forever

Shadow mode is a transition stage, not the destination.

Its purpose is to make enforcement safer, not to replace enforcement permanently.

### Mistake 4: Ignoring degradation paths

If every policy failure is treated as “allow everything in shadow, deny everything in prod,” you miss a major design opportunity.

Often the right outcome is not binary denial. It may be:

- switch to a smaller model
- disable a costly tool
- reduce concurrency
- move to read-only behavior
- end the run gracefully

Shadow mode should help design those paths too.

### Mistake 5: Looking only at averages

Average usage is not enough.

The important cases are often the long tail:

- the noisy tenant
- the runaway run
- the recursive workflow
- the bursty retry pattern

Those are the cases enforcement must handle well.

## How shadow mode supports trust

One of the hardest parts of introducing a new control layer is trust.

Application teams worry that governance will be too rigid.  
Platform teams worry that application teams will resist enforcement.  
Operators worry that a wrong policy will break production at the worst moment.

Shadow mode lowers that trust barrier.

It lets teams say:

- we are evaluating policy on real workloads
- we know what would have been denied
- we understand where the pressure points are
- we have tested the likely outcomes before turning on hard stops

That makes Cycles easier to adopt operationally.

## A strong first shadow policy

For many teams, a strong first rollout looks like this:

- evaluate **tenant budgets** in shadow mode
- evaluate **workflow budgets**, optionally keyed by run ID, in shadow mode
- instrument model calls and expensive tools
- log would-deny responses in the application
- compare estimates with actual usage from application telemetry
- review top offending runs and tenants
- add degradation rules before hard enforcement

This is a manageable first policy shape that gives useful signal quickly.

## When to leave shadow mode

A good rule is:

Move to hard enforcement when the shadow outcomes are no longer surprising.

That means:

- denial cases mostly match operator expectations
- estimates are directionally reliable
- major workflows have been observed sufficiently
- degradation paths are defined
- the team understands which scopes are responsible for decisions

When the policy is still producing confusing or obviously wrong results, stay in shadow and keep tuning.

When the policy starts behaving like the system you actually want, it is time to enforce.

## Summary

Shadow mode is how Cycles becomes operationally adoptable.

It lets teams evaluate real reservation and budget policy against real autonomous behavior without breaking production on day one.

That is critical because autonomous systems are hard to model perfectly in advance.

By using shadow mode, teams can:

- learn real consumption patterns
- tune tenant and workflow budgets, including per-execution workflow mappings
- refine estimate quality
- identify runaway behavior
- design degradation paths
- move to enforcement with far more confidence

That is how you roll out runtime authority safely.

## Next steps

To explore the Cycles stack:

- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)
- [AI Agent Cost Management: The Complete Guide](/blog/ai-agent-cost-management-guide) — the five-tier maturity model including shadow mode rollout



# Tenant Creation and Management in Cycles

Tenants are the top-level isolation boundary in Cycles. Every budget, API key, and reservation is scoped to exactly one tenant.

Before you can enforce budgets or issue API keys, you need at least one tenant. This guide covers the full tenant lifecycle through the Admin API. For an overview of how tenants fit into the broader scope and budget model, see [Understanding Tenants, Scopes, and Budgets](/how-to/understanding-tenants-scopes-and-budgets-in-cycles).

## What tenants are and when to create them

A tenant represents an isolated organizational unit in Cycles. Depending on your platform, a tenant might map to:

- a customer account in a SaaS product
- an internal team or department
- a business unit with its own budget
- a partner or reseller in a marketplace

Every API key belongs to one tenant. Every reservation is owned by one tenant. Every balance query is scoped to one tenant. This isolation is enforced at the protocol level — not by convention.

**Create a tenant when you need an independent budget boundary.** If two groups of users should not share budget, they should be separate tenants.

## Creating a tenant

Create a tenant using the Admin API with the `X-Admin-API-Key` header:

```bash
curl -s -X POST http://localhost:7979/v1/admin/tenants \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{
    "tenant_id": "acme-corp",
    "name": "Acme Corporation"
  }' | jq .
```

Response:

```json
{
  "tenant_id": "acme-corp",
  "name": "Acme Corporation",
  "status": "ACTIVE",
  "created_at": "2026-03-20T12:00:00Z"
}
```

### Tenant ID format

The `tenant_id` must be:

- **Lowercase alphanumeric with hyphens:** matches `^[a-z0-9-]+$`
- **Between 3 and 64 characters**
- **Kebab-case by convention:** for example, `acme-corp`, `demo-tenant`, `team-engineering`

Choose IDs that are stable and meaningful. The `tenant_id` is used in scope paths (e.g., `tenant:acme-corp/workspace:prod`), API key bindings, and audit logs. It cannot be changed after creation.

### Optional fields on creation

You can provide additional configuration when creating a tenant:

```bash
curl -s -X POST http://localhost:7979/v1/admin/tenants \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{
    "tenant_id": "acme-corp",
    "name": "Acme Corporation",
    "parent_tenant_id": "acme-group",
    "default_commit_overage_policy": "ALLOW_IF_AVAILABLE",
    "default_reservation_ttl_ms": 120000,
    "max_reservation_ttl_ms": 7200000,
    "max_reservation_extensions": 5,
    "reservation_expiry_policy": "AUTO_RELEASE",
    "metadata": {
      "billing_id": "cust_12345",
      "plan": "enterprise",
      "region": "us-east-1"
    }
  }' | jq .
```

The accepted optional fields on creation are:

| Field | Default | Description |
|---|---|---|
| `parent_tenant_id` | — | Parent tenant for hierarchical relationships (see [Hierarchical tenants](#hierarchical-tenants)) |
| `default_commit_overage_policy` | `ALLOW_IF_AVAILABLE` | Default overage policy: `REJECT`, `ALLOW_IF_AVAILABLE`, or `ALLOW_WITH_OVERDRAFT` |
| `default_reservation_ttl_ms` | `60000` (60s) | Default TTL when a reservation request does not specify `ttl_ms` |
| `max_reservation_ttl_ms` | `3600000` (1h) | Maximum allowed TTL; requests exceeding this are capped |
| `max_reservation_extensions` | `10` | Maximum TTL extensions per reservation |
| `reservation_expiry_policy` | `AUTO_RELEASE` | How expired reservations are handled: `AUTO_RELEASE`, `MANUAL_CLEANUP`, or `GRACE_ONLY`. **Creation-only** — it cannot be changed via `PATCH` afterwards |
| `metadata` | — | Key-value pairs for external references (up to 32 keys) |

Each of these fields is covered in detail in the sections below.

### Idempotent creation

Tenant creation is idempotent. If you retry a `POST /v1/admin/tenants` request with the same `tenant_id`:

- If the existing tenant has the **same name**, the server returns `200` with the existing tenant (not `201`).
- If the existing tenant has a **different name**, the server returns `409 CONFLICT`.

This makes it safe to retry tenant creation without checking whether the tenant already exists.

## Listing tenants

List all tenants with optional filters:

```bash
# List all active tenants
curl -s "http://localhost:7979/v1/admin/tenants?status=ACTIVE" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" | jq .
```

Response:

```json
{
  "tenants": [
    {
      "tenant_id": "acme-corp",
      "name": "Acme Corporation",
      "status": "ACTIVE",
      "created_at": "2026-03-20T12:00:00Z"
    }
  ],
  "has_more": false,
  "next_cursor": null
}
```

### Available filters

| Parameter | Description |
|---|---|
| `status` | Filter by status: `ACTIVE`, `SUSPENDED`, or `CLOSED` |
| `parent_tenant_id` | Filter by parent tenant (for hierarchical tenants) |
| `observe_mode` | Preview filter: `DISABLED`, `OBSERVE`, or `ENFORCE`; accepted but not applied by current v0.1.25.x admin servers |
| `search` | Case-insensitive substring match over `tenant_id` and `name` |
| `sort_by` / `sort_dir` | Sort key and direction (default: `created_at` descending) |
| `cursor` | Pagination cursor from a previous response |
| `limit` | Page size (default: 50, max: 100) |

See [Searching and Sorting Admin List Endpoints](/how-to/searching-and-sorting-admin-list-endpoints) for the shared `search`/`sort_by`/`sort_dir` parameter vocabulary.

### Cursor-based pagination

For large tenant lists, use cursor-based pagination:

```bash
# First page
curl -s "http://localhost:7979/v1/admin/tenants?limit=10" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" | jq .

# Next page (use next_cursor from previous response)
curl -s "http://localhost:7979/v1/admin/tenants?limit=10&cursor=eyJ0ZW5..." \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" | jq .
```

Continue until `has_more` is `false`.

## Retrieving a tenant

Get a single tenant by ID:

```bash
curl -s "http://localhost:7979/v1/admin/tenants/acme-corp" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" | jq .
```

This returns the full tenant object including all configuration, metadata, and timestamps.

## Updating a tenant

Update a tenant with `PATCH`:

```bash
curl -s -X PATCH http://localhost:7979/v1/admin/tenants/acme-corp \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{
    "name": "Acme Corp (Enterprise)",
    "metadata": {
      "billing_id": "cust_12345",
      "plan": "enterprise-plus"
    }
  }' | jq .
```

You can update:

- `name` — the display name
- `status` — transition between ACTIVE, SUSPENDED, and CLOSED (see lifecycle below)
- `metadata` — key-value pairs (replaces the full metadata object)
- `default_commit_overage_policy` — the default overage policy for all scopes
- `default_reservation_ttl_ms` — default TTL for reservations (1,000–86,400,000 ms)
- `max_reservation_ttl_ms` — maximum allowed TTL (1,000–86,400,000 ms)
- `max_reservation_extensions` — maximum TTL extensions per reservation (0+)

Fields not included in the `PATCH` request are left unchanged.

## Tenant status lifecycle

::: tip Status changes from the dashboard
Suspend, reactivate, and close are also one-click actions on the Tenants page in the [Cycles Admin Dashboard](/quickstart/deploying-the-cycles-dashboard). The Tenants list also supports **bulk suspend / reactivate** with a multi-select bar and per-tenant progress — useful when you need to lock down or restore many tenants at once during an incident.
:::

Every tenant has a status that controls what operations are allowed:

```
              suspend
  ACTIVE ─────────────► SUSPENDED
    │    ◄─────────────    │
    │       reactivate     │
    │                      │
    │   close              │   close
    ▼                      ▼
  CLOSED ◄────────────────────
```

### ACTIVE

The default state. All operations are allowed:

- New reservations can be created
- Existing reservations can be committed, released, or extended
- Balances can be queried
- New API keys can be issued

### SUSPENDED

A temporary block. Use this when you need to pause a tenant without permanent closure:

- **New reservations are blocked** — the server returns an error for any new reservation attempt
- Existing active reservations **can still be committed or released** — this prevents data loss from in-flight work
- Balances can still be queried
- The tenant can be **reactivated** back to ACTIVE at any time

**When to suspend:**

- A customer's payment has failed
- A security concern requires a temporary freeze
- An investigation is underway
- Usage needs to be paused during a plan change

```bash
# Suspend a tenant
curl -s -X PATCH http://localhost:7979/v1/admin/tenants/acme-corp \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{"status": "SUSPENDED"}' | jq .

# Reactivate the tenant
curl -s -X PATCH http://localhost:7979/v1/admin/tenants/acme-corp \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{"status": "ACTIVE"}' | jq .
```

### CLOSED

Permanent and irreversible. Use this only when a tenant is being decommissioned:

- All operations are blocked
- The tenant **cannot be reactivated**
- Data is retained for audit purposes
- **All owned objects cascade to terminal states automatically** (v0.1.25.35+)

**When to close:**

- A customer has churned and the account is being archived
- A test or demo tenant is no longer needed
- A department has been merged and its tenant is being retired

```bash
# Close a tenant (irreversible — triggers cascade)
curl -s -X PATCH http://localhost:7979/v1/admin/tenants/acme-corp \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{"status": "CLOSED"}' | jq .
```

::: warning
Closing a tenant is irreversible. If you need a temporary block, use SUSPENDED instead.
:::

#### What cascades automatically (v0.1.25.35+)

Pre-v0.1.25.35, closing a tenant was a pure status flip — operators then had to separately drain reservations, freeze budgets, revoke API keys, and disable webhooks by hand. Today, `cycles-server-admin` runs the cascade automatically and inline during the close (Mode B: the status flip commits first, then per-child terminal transitions complete before the response returns — not a single transaction):

| Owned object | Cascade action | Event kind |
|---|---|---|
| `BudgetLedger` | → `CLOSED` (final balance preserved for audit) | `budget.closed_via_tenant_cascade` |
| `ApiKey` | → `REVOKED` | `api_key.revoked_via_tenant_cascade` |
| Open `Reservation` | → `RELEASED` (reason `tenant_closed`, no overage debt) | `reservation.released_via_tenant_cascade` |
| `WebhookSubscription` | → `DISABLED` (re-enable blocked by Rule 2) | `webhook.disabled_via_tenant_cascade` |

The `*_via_tenant_cascade` identifiers are emitted as Event `event_type`s (declared in the governance spec's `EventType` enum since document revision v0.1.25.35, so `event_type=` filtering on them is spec-valid — though Event emission is SHOULD-level, so non-reference servers may not emit them). The matching **audit rows** are written as `operation="tenant_close_cascade"` with `resource_type`/`resource_id`. All four cascade **event rows** share a server-composed `correlation_id` (`tenant_close_cascade:<tenant_id>:<request_id>`; audit rows carry `request_id`/`trace_id`, not `correlation_id`) — you can find every side effect of a close with one events query:

```bash
# All cascade events for one close
curl -s "http://localhost:7979/v1/admin/events?correlation_id=<id>" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" | jq '.events[] | {event_type, data}'
```

After the cascade, mutating any owned object returns `409 TENANT_CLOSED` (Rule 2 — Terminal-Owner Mutation Guard). GET endpoints remain available for post-mortem audit reads. On the runtime plane, `cycles-server` 0.1.25.47+ enforces the same guard on persisting reservation create/commit/release/extend (runtime spec v0.1.25.13) — fresh dry-run and `/v1/decide` evaluations return `200 decision=DENY reason_code=TENANT_CLOSED` instead of a 409; runtime 0.1.25.46 and earlier surface closure there only as `401`s from revoked keys or `BUDGET_CLOSED`. Even on 0.1.25.47+, tenant-key runtime calls usually still fail `401` at the auth filter before the guard is reached — in practice the runtime 409 surfaces on admin-on-behalf-of release and in the post-flip/pre-revocation race window (see the [observability note](/protocol/tenant-close-cascade-semantics#what-the-runtime-plane-sees)). See [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics) for the full Rule 1 / Rule 2 contract and Mode A / Mode B semantics.

::: warning Don't pre-freeze before closing
On admin v0.1.25.35+, the cascade runs automatically and atomically from the operator's perspective (via Rule 2). **Do not** freeze budgets, revoke keys, or disable webhooks before closing — it's unnecessary, generates audit clutter, and (on future Mode A implementations) can cause cascades to roll back if a pre-freeze step fails. Just close the tenant; the cascade handles everything.

If you're running pre-v0.1.25.35 admin, cascade doesn't run — continue the manual cleanup until you upgrade.
:::

**Operator preview in the dashboard.** The [Cycles Admin Dashboard](/quickstart/deploying-the-cycles-dashboard) (v0.1.25.43+) shows what will be terminated in the CLOSE confirmation dialog before you click through — owned budgets, webhook subscriptions, API keys, and open reservations, with counts. Useful for estimating blast radius before pulling the trigger.

#### What to expect after close

Once you click close, the amber "Tenant closed — all owned objects are read-only." banner renders immediately on the tenant detail page. Behind the scenes, the behavior depends on versions:

| Your admin version | What happens |
|---|---|
| **v0.1.25.36+** (recommended) | Rule 1 cascade runs; Rule 2 guard active on every mutation endpoint. Budgets, API keys, reservations, and webhooks all reach terminal state automatically. Any admin-plane mutation attempt returns `409 TENANT_CLOSED`; runtime tenant-key mutations usually fail `401` at the auth filter first (see the runtime-plane caveat above). |
| **v0.1.25.35** | Rule 1 cascade runs; Rule 2 guard covers budget operations plus webhook create/update only. Policy, API key, remaining webhook, and bulk-action-row mutations against the now-closed tenant slip through silently until you upgrade (all completed in .36). Cascade itself still completes correctly. |
| **Pre-v0.1.25.35** | No cascade. Dashboard banner still renders (it's purely UI state), but owned objects stay in their pre-close state until you manually freeze / revoke / disable them. |

**Mode B timing.** runcycles' reference server uses Mode B (flip-first with guarded cascade) — `tenant.status` flips to `CLOSED` first, then children cascade inline. A GET against an owned budget in the milliseconds between flip and cascade-completion may still return the pre-terminal status, but any mutation will already be rejected by the Rule 2 guard. The observable window is typically sub-second on a healthy Redis; if it lingers longer, check the admin server's event-emission queue.

**Verify the cascade finished.** One audit query confirms every side effect:

```bash
# Audit rows carry request_id/trace_id (no correlation_id field); the
# cascade correlation_id is composed as tenant_close_cascade:<tenant_id>:<request_id>
RID=$(curl -s "http://localhost:7979/v1/admin/audit/logs?tenant_id=acme-corp&operation=updateTenant&limit=1" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" | jq -r '.logs[0].request_id')
CID="tenant_close_cascade:acme-corp:$RID"

# Pull every cascade event under that correlation_id
curl -s "http://localhost:7979/v1/admin/events?correlation_id=$CID" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" | jq '.events[] | {event_type, resource: .data}'
```

You should see one event per owned object — `budget.closed_via_tenant_cascade` per ledger, `api_key.revoked_via_tenant_cascade` per key, `webhook.disabled_via_tenant_cascade` per subscription — and one ledger-level `reservation.released_via_tenant_cascade` per closed budget that had `reserved > 0` (aggregate `released_amount`, not per-reservation). If the count is short: the runcycles server cascades inline before the PATCH response returns, so a shortfall there usually means a pre-v0.1.25.35 admin; on reconciler-based Mode B implementations the cascade may still be draining — wait a moment and re-query.

::: info Why tenants cannot be deleted
The admin API intentionally has no `DELETE /v1/admin/tenants/{tenant_id}` endpoint. Tenants are referenced by ID throughout the system — budgets, API keys, reservations, and audit logs all carry a `tenant_id`. Hard deletion would orphan these records and break audit trails.

`CLOSED` achieves the same operational goal: all operations are blocked and no new resources can be created. The difference is that the tenant record and all associated data remain queryable for reporting, compliance, and debugging.

**Cleaning up test tenants:** Use a naming convention like `test-*` or `demo-*` and batch-close them with `bulkActionTenants`:

```bash
curl -X POST http://localhost:7979/v1/admin/tenants/bulk-action \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "CLOSE",
    "filter": { "search": "test-" },
    "idempotency_key": "cleanup-2026-04-20-test-tenants"
  }' | jq '{succeeded: (.succeeded | length), failed: .failed}'
```

The cascade runs per-row — each closed tenant's owned objects terminate under its own `correlation_id`. Rows already in the target state (e.g., a tenant that is already CLOSED) land in `skipped[]` with `reason: "ALREADY_IN_TARGET_STATE"`; genuine failures land in `failed[]` with a per-row `error_code` and `message`. The rest of the batch proceeds either way.

The data footprint of a closed tenant is minimal.
:::

### Invalid transitions

The server rejects invalid status transitions with `400 INVALID_REQUEST`:

- `CLOSED → ACTIVE` (cannot reactivate a closed tenant)
- `CLOSED → SUSPENDED` (cannot suspend a closed tenant)

## Configuring tenant defaults

Each tenant has configuration that governs how reservations behave. These properties are set at creation; all except `reservation_expiry_policy` can also be updated later via `PATCH`.

### Settable per tenant

| Property | Default | Description |
|---|---|---|
| `default_commit_overage_policy` | `ALLOW_IF_AVAILABLE` | What happens when actual spend exceeds the reserved amount |
| `default_reservation_ttl_ms` | `60000` (60s) | Default TTL when a reservation request does not specify `ttl_ms` |
| `max_reservation_ttl_ms` | `3600000` (1h) | Maximum allowed TTL; requests exceeding this are capped |
| `max_reservation_extensions` | `10` | Maximum TTL extensions per reservation (prevents zombie reservations) |
| `reservation_expiry_policy` | `AUTO_RELEASE` | How expired reservations are handled. **Creation-only** — not updatable via `PATCH` |

All of these except `reservation_expiry_policy` can also be updated later via `PATCH /v1/admin/tenants/{tenant_id}`. The expiry policy is fixed at creation; to change it you would need a new tenant.

### Commit overage policies

The `default_commit_overage_policy` controls what happens when a commit's `actual` amount exceeds the originally reserved `estimate`:

| Policy | Behavior |
|---|---|
| `REJECT` | Fail the commit if actual > reserved |
| `ALLOW_IF_AVAILABLE` | Charge the delta from remaining budget if sufficient |
| `ALLOW_WITH_OVERDRAFT` | Create debt up to the scope's `overdraft_limit` if budget is insufficient |

Set this at the tenant level to establish a baseline, then override per-budget-ledger or per-reservation as needed.

```bash
curl -s -X PATCH http://localhost:7979/v1/admin/tenants/acme-corp \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{"default_commit_overage_policy": "ALLOW_IF_AVAILABLE"}' | jq .
```

### Reservation expiry policies

The `reservation_expiry_policy` controls what happens when a reservation exceeds its TTL without being committed or released:

| Policy | Behavior |
|---|---|
| `AUTO_RELEASE` | Expired reservations are automatically released after a grace period, freeing the reserved budget |
| `MANUAL_CLEANUP` | Expired reservations require explicit release or a cleanup job |
| `GRACE_ONLY` | Allow commits during the grace period, then mark the reservation as `EXPIRED` |

For most deployments, `AUTO_RELEASE` is the safest default — it prevents zombie reservations from permanently locking budget.

### TTL configuration

- **`default_reservation_ttl_ms`** sets the TTL used when a reservation request does not specify `ttl_ms`. A value of 60,000 ms (60 seconds) works well for synchronous LLM calls. Increase it for longer-running workflows.

- **`max_reservation_ttl_ms`** caps the maximum TTL any reservation can request. This prevents callers from holding budget indefinitely. Requests that specify a `ttl_ms` exceeding this value are silently capped.

- **`max_reservation_extensions`** limits how many times a reservation's TTL can be extended. This prevents zombie reservations from being extended forever. A value of 10 is generous for most use cases.

Configure TTL settings per tenant:

```bash
curl -s -X PATCH http://localhost:7979/v1/admin/tenants/acme-corp \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{
    "default_reservation_ttl_ms": 120000,
    "max_reservation_ttl_ms": 7200000,
    "max_reservation_extensions": 5
  }' | jq .
```

## Hierarchical tenants

Cycles supports parent-child tenant relationships using the `parent_tenant_id` field. This enables:

- **Organizational hierarchy:** A parent company with subsidiary business units
- **Reseller models:** A partner who manages multiple end-customer tenants
- **Budget delegation:** A parent tenant that distributes budget to child tenants

### Creating a hierarchy

```bash
# Create the parent tenant
curl -s -X POST http://localhost:7979/v1/admin/tenants \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{
    "tenant_id": "acme-group",
    "name": "Acme Group (Parent)"
  }' | jq .

# Create child tenants under the parent
curl -s -X POST http://localhost:7979/v1/admin/tenants \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{
    "tenant_id": "acme-engineering",
    "name": "Acme Engineering",
    "parent_tenant_id": "acme-group"
  }' | jq .

curl -s -X POST http://localhost:7979/v1/admin/tenants \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{
    "tenant_id": "acme-marketing",
    "name": "Acme Marketing",
    "parent_tenant_id": "acme-group"
  }' | jq .
```

### Listing child tenants

```bash
curl -s "http://localhost:7979/v1/admin/tenants?parent_tenant_id=acme-group" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" | jq .
```

### How hierarchical tenants work

Each child tenant is still a fully independent isolation boundary:

- Child tenants have their own budgets, API keys, and reservations
- A child tenant's API key cannot access the parent tenant's resources (and vice versa)
- Budget is not automatically shared or aggregated between parent and child

The `parent_tenant_id` relationship is useful for:

- **Consolidated billing:** Query all child tenants under a parent for billing reports
- **Administrative grouping:** List and manage related tenants together
- **Organizational modeling:** Reflect your real-world structure in the tenant hierarchy

## Tenant metadata

Each tenant supports a `metadata` field — a map of up to 32 key-value pairs for storing arbitrary information:

```bash
curl -s -X POST http://localhost:7979/v1/admin/tenants \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{
    "tenant_id": "acme-corp",
    "name": "Acme Corporation",
    "metadata": {
      "billing_id": "cust_12345",
      "plan": "enterprise",
      "region": "us-east-1",
      "owner_email": "admin@acme.com",
      "stripe_customer_id": "cus_abc123"
    }
  }' | jq .
```

Common metadata patterns:

| Key | Purpose |
|---|---|
| `billing_id` | Link to your billing system's customer ID |
| `plan` | Subscription tier (free, pro, enterprise) |
| `region` | Geographic region for data residency |
| `owner_email` | Primary contact for the tenant |
| `external_id` | ID from your own system for correlation |

::: info
Updating metadata replaces the entire metadata object. To add a new key while keeping existing ones, include all keys in the update.
:::

## End-to-end: onboarding a new tenant

Here is the complete sequence to go from zero to a working tenant with budget enforcement:

### Step 1: Create the tenant

```bash
curl -s -X POST http://localhost:7979/v1/admin/tenants \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{
    "tenant_id": "acme-corp",
    "name": "Acme Corporation",
    "metadata": {"plan": "pro"}
  }' | jq .
```

### Step 2: Create an API key for the tenant

```bash
API_KEY=$(curl -s -X POST http://localhost:7979/v1/admin/api-keys \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{
    "tenant_id": "acme-corp",
    "name": "production-key",
    "permissions": [
      "reservations:create",
      "reservations:commit",
      "reservations:release",
      "reservations:extend",
      "reservations:list",
      "balances:read",
      "budgets:read",
      "budgets:write"
    ]
  }' | jq -r '.key_secret')

echo "API Key: $API_KEY"
```

::: warning Explicit permissions replace the defaults
An explicit `permissions` array **replaces** the default set — it is not merged with it. A key created with only the six runtime permissions cannot call the budget endpoints in Step 3 (`POST /v1/admin/budgets` requires `budgets:write`). Either include `budgets:read`/`budgets:write` as shown, or omit `permissions` entirely to get the 10-permission default set, which includes them.
:::

Save this key — the full secret is only returned once. See [API Key Management](/how-to/api-key-management-in-cycles) for rotation and security practices.

### Step 3: Create a budget for the tenant

```bash
curl -s -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $API_KEY" \
  -d '{
    "scope": "tenant:acme-corp",
    "unit": "USD_MICROCENTS",
    "allocated": {"amount": 100000000, "unit": "USD_MICROCENTS"}
  }' | jq .
```

This creates a budget of $1.00 (100,000,000 microcents). See [Budget Allocation and Management](/how-to/budget-allocation-and-management-in-cycles) for funding patterns and hierarchical budgets.

### Step 4: Make the first reservation

```bash
RESERVATION_ID=$(curl -s -X POST http://localhost:7878/v1/reservations \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $API_KEY" \
  -d '{
    "idempotency_key": "onboard-test-001",
    "subject": {"tenant": "acme-corp"},
    "action": {"kind": "llm.completion", "name": "openai:gpt-4o"},
    "estimate": {"amount": 500000, "unit": "USD_MICROCENTS"},
    "ttl_ms": 30000
  }' | jq -r '.reservation_id')

echo "Reserved: $RESERVATION_ID"
```

If you see `"decision": "ALLOW"`, the tenant is fully operational.

### Step 5: Commit and verify

```bash
# Commit actual spend
curl -s -X POST "http://localhost:7878/v1/reservations/$RESERVATION_ID/commit" \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $API_KEY" \
  -d '{
    "idempotency_key": "onboard-commit-001",
    "actual": {"amount": 350000, "unit": "USD_MICROCENTS"}
  }' | jq .

# Check the balance
curl -s "http://localhost:7878/v1/balances?tenant=acme-corp" \
  -H "X-Cycles-API-Key: $API_KEY" | jq .
```

The tenant is now live with budget enforcement.

## Common use cases

### SaaS per-customer isolation

Each customer gets their own tenant with an independent budget:

```bash
# Customer onboarding script
for customer in "startup-co" "bigcorp-inc" "agency-xyz"; do
  curl -s -X POST http://localhost:7979/v1/admin/tenants \
    -H "Content-Type: application/json" \
    -H "X-Admin-API-Key: $ADMIN_API_KEY" \
    -d "{
      \"tenant_id\": \"$customer\",
      \"name\": \"$customer\"
    }"
done
```

This ensures one customer's runaway agent cannot consume another customer's budget. See [Multi-Tenant AI Cost Control](/blog/multi-tenant-ai-cost-control-per-tenant-budgets-quotas-isolation) for the full pattern.

### Internal department budgets

Use tenants to give each department its own spending boundary:

```bash
# Engineering gets a larger budget than marketing
# Create tenants under a parent
for dept in "eng" "marketing" "support"; do
  curl -s -X POST http://localhost:7979/v1/admin/tenants \
    -H "Content-Type: application/json" \
    -H "X-Admin-API-Key: $ADMIN_API_KEY" \
    -d "{
      \"tenant_id\": \"dept-$dept\",
      \"name\": \"Department: $dept\",
      \"parent_tenant_id\": \"company-hq\"
    }"
done
```

### Partner and reseller hierarchies

A reseller manages multiple end-customer tenants:

```bash
# Reseller as parent
curl -s -X POST http://localhost:7979/v1/admin/tenants \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{
    "tenant_id": "reseller-alpha",
    "name": "Alpha Partners",
    "metadata": {"type": "reseller", "commission_rate": "15"}
  }' | jq .

# End customers under the reseller
curl -s -X POST http://localhost:7979/v1/admin/tenants \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{
    "tenant_id": "alpha-customer-1",
    "name": "Customer One",
    "parent_tenant_id": "reseller-alpha"
  }' | jq .
```

### Environment separation

Use tenants to isolate production from staging and development:

```bash
for env in "acme-prod" "acme-staging" "acme-dev"; do
  curl -s -X POST http://localhost:7979/v1/admin/tenants \
    -H "Content-Type: application/json" \
    -H "X-Admin-API-Key: $ADMIN_API_KEY" \
    -d "{
      \"tenant_id\": \"$env\",
      \"name\": \"Acme ($env)\"
    }"
done
```

Give production a large budget and dev a small one. A bug in staging cannot drain the production budget.

## Best practices

### Naming conventions

Use stable, semantic tenant IDs that reflect your domain:

| Good | Avoid |
|---|---|
| `customer-acme` | `cust_12345` (opaque database ID) |
| `dept-engineering` | `eng` (too short, ambiguous) |
| `partner-alpha` | `PARTNER_ALPHA` (must be lowercase) |

Tenant IDs appear in scope paths (`tenant:customer-acme/workspace:prod`), audit logs, and API key bindings. Choose names that are readable and meaningful to your team.

### One tenant = one isolation boundary

Do not multiplex unrelated customers or teams into a single tenant. If two groups should not share budget, they need separate tenants. Use [hierarchical tenants](#hierarchical-tenants) to model organizational relationships rather than sharing a single tenant.

### Suspend before you close

Use `SUSPENDED` for temporary blocks — payment failures, security investigations, plan changes. A suspended tenant can be reactivated at any time.

Use `CLOSED` only for permanent decommission. It is irreversible. If there is any chance you will need the tenant again, use `SUSPENDED`.

### Use metadata consistently

Pick a standard set of metadata keys and use them across all tenants. This makes it easy to query and correlate tenant data with external systems:

```json
{
  "billing_id": "cust_12345",
  "plan": "enterprise",
  "region": "us-east-1",
  "owner_email": "admin@acme.com"
}
```

### Set overage policy at the tenant level

The `default_commit_overage_policy` establishes a baseline for all scopes under the tenant. The default is `ALLOW_IF_AVAILABLE`, which caps charges to available budget and never creates debt. Switch to `REJECT` for hard stops, or `ALLOW_WITH_OVERDRAFT` when exact accounting with debt is needed.

Override the policy per-budget-ledger or per-reservation for specific scopes that need different behavior.

### Create API keys per environment

Issue separate API keys for production, staging, and development — even within the same tenant. This makes it easy to revoke one environment's access without affecting others. See [API Key Management](/how-to/api-key-management-in-cycles) for rotation practices.

### Automate tenant onboarding

The create tenant → create API key → create budget sequence should be scripted, not manual. This ensures consistency, reduces errors, and makes it easy to onboard new customers at scale.

```bash
# Example: onboard a new customer
TENANT_ID="customer-${CUSTOMER_SLUG}"
curl -s -X POST http://localhost:7979/v1/admin/tenants \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d "{\"tenant_id\": \"$TENANT_ID\", \"name\": \"$CUSTOMER_NAME\"}"

API_KEY=$(curl -s -X POST http://localhost:7979/v1/admin/api-keys \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d "{\"tenant_id\": \"$TENANT_ID\", \"name\": \"prod-key\", \"permissions\": [\"reservations:create\",\"reservations:commit\",\"reservations:release\",\"balances:read\",\"budgets:read\",\"budgets:write\"]}" \
  | jq -r '.key_secret')

# budgets:read/budgets:write are required for the budget call below —
# an explicit permissions array REPLACES the defaults, it is not merged.
# Alternatively, omit "permissions" to get the default set (which includes them).

curl -s -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $API_KEY" \
  -d "{\"scope\": \"tenant:$TENANT_ID\", \"unit\": \"USD_MICROCENTS\", \"allocated\": {\"amount\": $BUDGET_AMOUNT, \"unit\": \"USD_MICROCENTS\"}}"
```

## Troubleshooting

### TENANT_NOT_FOUND

The tenant does not exist. Create it first with `POST /v1/admin/tenants`.

This also occurs when creating an API key for a non-existent tenant — the tenant must exist before you can issue keys for it.

### TENANT_SUSPENDED

The tenant's status is `SUSPENDED`. New reservations are blocked.

To resume operations, reactivate the tenant:

```bash
curl -s -X PATCH http://localhost:7979/v1/admin/tenants/acme-corp \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{"status": "ACTIVE"}' | jq .
```

### TENANT_CLOSED

The tenant has been permanently closed. This cannot be reversed. If you need a new tenant, create one with a different `tenant_id`.

Returned by every mutating admin-plane operation on the closed tenant's owned objects (admin v0.1.25.35+, full coverage v0.1.25.36+) and, since `cycles-server` 0.1.25.47, by persisting reservation create/commit/release/extend on the runtime plane (runtime spec v0.1.25.13).

### 403 FORBIDDEN (tenant mismatch)

The `subject.tenant` in your request does not match the effective tenant derived from the API key.

Check:
1. The `X-Cycles-API-Key` header is for the correct tenant
2. The `subject.tenant` field matches the API key's tenant
3. Use the `X-Cycles-Tenant` response header (if present) to see which tenant the server resolved

See [Authentication, Tenancy, and API Keys](/protocol/authentication-tenancy-and-api-keys-in-cycles) for the full authentication model.

### 409 CONFLICT on tenant creation

You tried to create a tenant with a `tenant_id` that already exists but with a different `name`. Either:
- Use the existing tenant as-is
- Choose a different `tenant_id`

### Common mistakes

**Creating budgets before tenants.** The tenant must exist before you can create API keys or budgets for it. Follow the onboarding sequence: tenant → API key → budget.

**Using the wrong auth header.** Tenant management uses `X-Admin-API-Key` (system admin). Budget and reservation operations use `X-Cycles-API-Key` (tenant-scoped). See the [Architecture Overview](/quickstart/architecture-overview-how-cycles-fits-together) for which header to use where.

**Closing tenants prematurely.** Use `SUSPENDED` for temporary blocks. Only use `CLOSED` when the tenant is being permanently decommissioned.

## Next steps

- [Tenants, Scopes, and Budgets](/how-to/understanding-tenants-scopes-and-budgets-in-cycles) — how tenants, scopes, and budgets work together
- [API Key Management](/how-to/api-key-management-in-cycles) — create and rotate API keys for your tenants
- [Budget Allocation and Management](/how-to/budget-allocation-and-management-in-cycles) — set up budgets at tenant and sub-scopes
- [Tenant, Workflow, and Run Budgets](/how-to/how-to-model-tenant-workflow-and-run-budgets-in-cycles) — design multi-level budget policies
- [Authentication, Tenancy, and API Keys](/protocol/authentication-tenancy-and-api-keys-in-cycles) — how tenant isolation is enforced at the protocol level
- [Scope Derivation](/protocol/how-scope-derivation-works-in-cycles) — how tenant scopes fit into the budget hierarchy
- [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack) — set up the Cycles infrastructure from scratch

## Related concepts

- [Tenant lifecycle cascade semantics](/blog/tenant-lifecycle-cascade-semantics-at-scale)
- [Audit trail as a runtime-authority byproduct](/blog/runtime-authority-byproducts-audit-trail-and-attribution-by-default)
- [Agent delegation chains and authority attenuation](/blog/agent-delegation-chains-authority-attenuation-not-trust-propagation)



# Testing with Cycles

This guide covers how to test code that uses the `@cycles` decorator (Python), the `@Cycles` annotation (Java), or the `withCycles` HOF (TypeScript) and the `CyclesClient` interface.

::: tip Key principle
Test business logic separately from budget enforcement. Mock the Cycles client in unit tests and use a real Cycles server in integration tests.
:::

## Python

### Unit testing @cycles-decorated functions

The `@cycles` decorator requires a client to function. In a unit test, you can test business logic by calling the underlying function directly without the decorator, or by mocking the client.

For plain function logic (without budget enforcement), test the function directly:

```python
def test_business_logic():
    result = call_llm("some text")
    assert result == "expected output"
```

### Mocking CyclesClient with pytest

When testing code that uses `CyclesClient` programmatically, mock the client responses:

```python
from unittest.mock import MagicMock, ANY
from runcycles import CyclesClient, CyclesResponse
import pytest

def test_successful_processing():
    client = MagicMock(spec=CyclesClient)

    # Mock reservation response
    client.create_reservation.return_value = CyclesResponse.success(200, {
        "reservation_id": "res-123",
        "decision": "ALLOW",
        "expires_at_ms": 1709312345678,
    })

    # Mock commit response
    client.commit_reservation.return_value = CyclesResponse.success(200, {
        "status": "COMMITTED",
    })

    result = process_document(client, "doc-1", "content")

    assert result is not None
    client.create_reservation.assert_called_once()
    client.commit_reservation.assert_called_once()


def test_budget_denied():
    client = MagicMock(spec=CyclesClient)

    # Insufficient budget returns 409
    client.create_reservation.return_value = CyclesResponse.http_error(
        409, "Insufficient remaining balance",
        body={"error": "BUDGET_EXCEEDED", "message": "Insufficient remaining balance"},
    )

    result = process_document(client, "doc-1", "content")

    assert result == "Budget exhausted. Please try again later."
    client.commit_reservation.assert_not_called()


def test_release_on_failure():
    client = MagicMock(spec=CyclesClient)

    client.create_reservation.return_value = CyclesResponse.success(200, {
        "reservation_id": "res-123",
        "decision": "ALLOW",
    })

    with pytest.raises(RuntimeError):
        process_document_that_fails(client, "doc-1", "content")

    # Verify budget was released
    client.release_reservation.assert_called_once()
```

### Testing with pytest-httpx

For integration-style tests, use `pytest-httpx` to mock HTTP responses:

```python
from runcycles import CyclesClient, CyclesConfig

def test_full_lifecycle(httpx_mock):
    httpx_mock.add_response(
        method="POST",
        url="http://localhost:7878/v1/reservations",
        json={
            "reservation_id": "res-test-001",
            "decision": "ALLOW",
            "expires_at_ms": 1709312345678,
            "affected_scopes": ["tenant:test"],
        },
        status_code=200,
    )

    httpx_mock.add_response(
        method="POST",
        url="http://localhost:7878/v1/reservations/res-test-001/commit",
        json={"status": "COMMITTED"},
        status_code=200,
    )

    config = CyclesConfig(base_url="http://localhost:7878", api_key="test-key")
    request = {
        "idempotency_key": "test-001",
        "subject": {"tenant": "test"},
        "action": {"kind": "test", "name": "integration"},
        "estimate": {"unit": "USD_MICROCENTS", "amount": 100},
    }
    with CyclesClient(config) as client:
        response = client.create_reservation(request)
        assert response.is_success
        assert response.get_body_attribute("reservation_id") == "res-test-001"
```

### Testing error handling

```python
from runcycles import BudgetExceededError, CyclesProtocolError

def test_budget_exceeded_handling():
    ex = BudgetExceededError(
        "Budget exceeded",
        status=409,
        error_code="BUDGET_EXCEEDED",
    )
    assert ex.is_budget_exceeded()
    assert not ex.is_reservation_expired()
    assert ex.status == 409

def test_retry_after_handling():
    ex = CyclesProtocolError(
        "Try again later",
        status=409,
        error_code="BUDGET_EXCEEDED",
        retry_after_ms=5000,
    )
    assert ex.retry_after_ms == 5000
```

### Testing async code

```python
import pytest
from runcycles import AsyncCyclesClient, CyclesConfig

@pytest.mark.asyncio
async def test_async_reservation(httpx_mock):
    httpx_mock.add_response(
        method="POST",
        url="http://localhost:7878/v1/reservations",
        json={"reservation_id": "res-async-001", "decision": "ALLOW"},
        status_code=200,
    )

    config = CyclesConfig(base_url="http://localhost:7878", api_key="test-key")
    request = {
        "idempotency_key": "test-async-001",
        "subject": {"tenant": "test"},
        "action": {"kind": "test", "name": "integration"},
        "estimate": {"unit": "USD_MICROCENTS", "amount": 100},
    }
    async with AsyncCyclesClient(config) as client:
        response = await client.create_reservation(request)
        assert response.is_success
```

## Java (Spring)

### Unit testing @Cycles-annotated methods

The `@Cycles` annotation is driven by Spring AOP. In a plain unit test (without Spring context), the annotation has no effect — the method runs normally without any reservation lifecycle.

This means you can unit test the method's business logic without Cycles getting involved:

```java
@Test
void testBusinessLogic() {
    LlmService service = new LlmService(mockChatModel);
    String result = service.summarize("some text");
    assertEquals("expected output", result);
}
```

No mocking of Cycles is needed for pure unit tests.

### Mocking CyclesClient

When testing code that uses `CyclesClient` programmatically, mock the client:

```java
@ExtendWith(MockitoExtension.class)
class DocumentProcessorTest {

    @Mock
    private CyclesClient cyclesClient;

    @InjectMocks
    private DocumentProcessor processor;

    @Test
    void testSuccessfulProcessing() {
        Map<String, Object> reserveBody = Map.of(
            "reservation_id", "res-123",
            "decision", "ALLOW",
            "expires_at_ms", System.currentTimeMillis() + 60000
        );
        when(cyclesClient.createReservation(any()))
            .thenReturn(CyclesResponse.success(200, reserveBody));

        Map<String, Object> commitBody = Map.of("status", "COMMITTED");
        when(cyclesClient.commitReservation(eq("res-123"), any()))
            .thenReturn(CyclesResponse.success(200, commitBody));

        String result = processor.processDocument("doc-1", "content");

        assertNotNull(result);
        verify(cyclesClient).createReservation(any());
        verify(cyclesClient).commitReservation(eq("res-123"), any());
    }

    @Test
    void testBudgetDenied() {
        when(cyclesClient.createReservation(any()))
            .thenReturn(CyclesResponse.httpError(409, "Insufficient remaining balance",
                Map.of("error", "BUDGET_EXCEEDED",
                       "message", "Insufficient remaining balance")));

        String result = processor.processDocument("doc-1", "content");

        assertEquals("Budget exhausted. Please try again later.", result);
        verify(cyclesClient, never()).commitReservation(any(), any());
    }

    @Test
    void testReleaseOnFailure() {
        Map<String, Object> reserveBody = Map.of(
            "reservation_id", "res-123",
            "decision", "ALLOW"
        );
        when(cyclesClient.createReservation(any()))
            .thenReturn(CyclesResponse.success(200, reserveBody));

        doThrow(new RuntimeException("LLM error"))
            .when(mockLlm).call(any());

        assertThrows(RuntimeException.class,
            () -> processor.processDocument("doc-1", "content"));

        verify(cyclesClient).releaseReservation(eq("res-123"), any());
    }
}
```

### Integration testing with the @Cycles annotation

To test the full `@Cycles` lifecycle in a Spring context, mock the `CyclesClient` bean:

```java
@SpringBootTest
class CyclesIntegrationTest {

    @MockBean
    private CyclesClient cyclesClient;

    @Autowired
    private LlmService llmService;

    @Test
    void testAnnotatedMethodWithAllow() {
        Map<String, Object> reserveBody = Map.of(
            "reservation_id", "res-test-001",
            "decision", "ALLOW",
            "expires_at_ms", System.currentTimeMillis() + 60000,
            "affected_scopes", List.of("tenant:test"),
            "scope_path", "tenant:test",
            "reserved", Map.of("amount", 5000, "unit", "USD_MICROCENTS")
        );
        when(cyclesClient.createReservation(any()))
            .thenReturn(CyclesResponse.success(200, reserveBody));

        Map<String, Object> commitBody = Map.of(
            "status", "COMMITTED",
            "charged", Map.of("amount", 3200, "unit", "USD_MICROCENTS")
        );
        when(cyclesClient.commitReservation(any(), any()))
            .thenReturn(CyclesResponse.success(200, commitBody));

        String result = llmService.summarize("test input");

        assertNotNull(result);
        verify(cyclesClient).createReservation(any());
        verify(cyclesClient).commitReservation(eq("res-test-001"), any());
    }

    @Test
    void testAnnotatedMethodWithDeny() {
        // A live (non-dry-run) denial is a 409 BUDGET_EXCEEDED error body —
        // there is no "decision" field on 4xx responses
        Map<String, Object> denyBody = Map.of(
            "error", "BUDGET_EXCEEDED",
            "message", "Insufficient budget"
        );
        when(cyclesClient.createReservation(any()))
            .thenReturn(CyclesResponse.httpError(409, "Insufficient budget", denyBody));

        assertThrows(CyclesProtocolException.class,
            () -> llmService.summarize("test input"));
    }
}
```

### Integration testing with a real Cycles server

For end-to-end tests, use Testcontainers to spin up Redis and the Cycles server (`ghcr.io/runcycles/cycles-server`) on a shared network:

```java
@SpringBootTest
@Testcontainers
class FullIntegrationTest {

    static Network network = Network.newNetwork();

    @Container
    static GenericContainer<?> redis = new GenericContainer<>("redis:7-alpine")
        .withNetwork(network)
        .withNetworkAliases("redis")
        .withExposedPorts(6379);

    @Container
    static GenericContainer<?> cyclesServer =
        new GenericContainer<>("ghcr.io/runcycles/cycles-server:latest")
            .withNetwork(network)
            .withEnv("REDIS_HOST", "redis")
            .withEnv("REDIS_PORT", "6379")
            .withExposedPorts(7878)
            .dependsOn(redis);

    @DynamicPropertySource
    static void configureProperties(DynamicPropertyRegistry registry) {
        registry.add("cycles.base-url", () ->
            "http://" + cyclesServer.getHost() + ":" + cyclesServer.getMappedPort(7878));
        registry.add("cycles.api-key", () -> "test-key");
        registry.add("cycles.tenant", () -> "test-tenant");
    }

    @Autowired
    private CyclesClient cyclesClient;

    @Test
    void testFullLifecycle() {
        ReservationCreateRequest request = ReservationCreateRequest.builder()
            .idempotencyKey("integration-test-001")
            .subject(Subject.builder().tenant("test-tenant").build())
            .action(new Action("test", "integration", null))
            .estimate(new Amount(Unit.USD_MICROCENTS, 100L))
            .build();

        CyclesResponse<Map<String, Object>> response =
            cyclesClient.createReservation(request);

        assertTrue(response.is2xx());
    }
}
```

::: warning Provision test data first
The server authenticates `X-Cycles-API-Key` against keys stored in Redis and enforces budgets seeded there. Seed a test API key and a budget for `test-tenant` (for example via the Cycles Admin API, or the seeding scripts in the [cycles-server repo](https://github.com/runcycles/cycles-server)) before the reservation assertion above will pass.
:::

### Testing CyclesFieldResolver implementations

Test custom field resolvers directly:

```java
@Test
void testTenantResolver() {
    RepositoryAccessService repoService = mock(RepositoryAccessService.class);
    when(repoService.findTenant()).thenReturn(Optional.of("resolved-tenant"));

    CyclesTenantResolver resolver = new CyclesTenantResolver();
    ReflectionTestUtils.setField(resolver, "repositoryAccessService", repoService);

    assertEquals("resolved-tenant", resolver.resolve());
}
```

### Testing SpEL expressions

Test that your SpEL expressions evaluate correctly:

```java
@Test
void testEstimateExpression() throws NoSuchMethodException {
    CyclesExpressionEvaluator evaluator = new CyclesExpressionEvaluator();

    Method method = LlmService.class.getMethod("generate", int.class);
    Object[] args = { 500 };

    long result = evaluator.evaluate("#p0 * 10", method, args, null, null);
    assertEquals(5000, result);
}
```

## TypeScript

### Unit testing withCycles-wrapped functions

`withCycles` wraps a function with budget governance. In a unit test, you can test the inner function directly without the wrapper:

```typescript
// The inner function (no budget governance)
async function callLlm(prompt: string): Promise<string> {
  return `Response to: ${prompt}`;
}

// Test the business logic directly
import { describe, it, expect } from "vitest";

describe("callLlm", () => {
  it("returns a response", async () => {
    const result = await callLlm("Hello");
    expect(result).toBe("Response to: Hello");
  });
});
```

### Mocking CyclesClient with Vitest

When testing code that uses `CyclesClient` programmatically, mock the client methods:

```typescript
import { describe, it, expect, vi } from "vitest";
import { CyclesClient } from "runcycles";
import { CyclesResponse } from "runcycles";

describe("processDocument", () => {
  it("creates reservation and commits on success", async () => {
    const client = {
      config: { tenant: "acme" },
      createReservation: vi.fn().mockResolvedValue(
        CyclesResponse.success(200, {
          reservation_id: "res-123",
          decision: "ALLOW",
          affected_scopes: ["tenant:acme"],
          expires_at_ms: Date.now() + 60000,
        }),
      ),
      commitReservation: vi.fn().mockResolvedValue(
        CyclesResponse.success(200, { status: "COMMITTED" }),
      ),
      releaseReservation: vi.fn(),
      extendReservation: vi.fn(),
    };

    const result = await processDocument(client as any, "doc-1", "content");

    expect(result).toBeDefined();
    expect(client.createReservation).toHaveBeenCalledOnce();
    expect(client.commitReservation).toHaveBeenCalledOnce();
  });

  it("returns fallback on budget denied", async () => {
    const client = {
      config: { tenant: "acme" },
      createReservation: vi.fn().mockResolvedValue(
        CyclesResponse.httpError(
          409,
          "Insufficient remaining balance",
          { error: "BUDGET_EXCEEDED", message: "Insufficient remaining balance" },
        ),
      ),
      commitReservation: vi.fn(),
      releaseReservation: vi.fn(),
      extendReservation: vi.fn(),
    };

    const result = await processDocument(client as any, "doc-1", "content");

    expect(result).toBe("Budget exhausted. Please try again later.");
    expect(client.commitReservation).not.toHaveBeenCalled();
  });

  it("releases reservation on processing failure", async () => {
    const client = {
      config: { tenant: "acme" },
      createReservation: vi.fn().mockResolvedValue(
        CyclesResponse.success(200, {
          reservation_id: "res-123",
          decision: "ALLOW",
          affected_scopes: [],
        }),
      ),
      commitReservation: vi.fn(),
      releaseReservation: vi.fn().mockResolvedValue(
        CyclesResponse.success(200, { status: "RELEASED" }),
      ),
      extendReservation: vi.fn(),
    };

    await expect(
      processDocumentThatFails(client as any, "doc-1", "content"),
    ).rejects.toThrow();

    expect(client.releaseReservation).toHaveBeenCalledOnce();
  });
});
```

### Mocking fetch for integration tests

For integration-style tests, mock the global `fetch` to return Cycles API responses:

```typescript
import { describe, it, expect, vi, afterEach } from "vitest";
import { CyclesClient, CyclesConfig } from "runcycles";

function mockFetchSequence(
  responses: Array<{ status: number; body: Record<string, unknown> }>,
) {
  let callIndex = 0;
  vi.stubGlobal(
    "fetch",
    vi.fn().mockImplementation(() => {
      const resp = responses[callIndex] ?? responses[responses.length - 1];
      callIndex++;
      return Promise.resolve({
        status: resp.status,
        statusText: resp.status >= 400 ? "Error" : "OK",
        json: () => Promise.resolve(resp.body),
        headers: new Headers(),
      });
    }),
  );
}

describe("full lifecycle", () => {
  afterEach(() => {
    vi.unstubAllGlobals();
  });

  it("reserves, executes, and commits", async () => {
    mockFetchSequence([
      {
        status: 200,
        body: {
          decision: "ALLOW",
          reservation_id: "res-test-001",
          affected_scopes: ["tenant:test"],
          expires_at_ms: Date.now() + 60000,
        },
      },
      { status: 200, body: { status: "COMMITTED" } },
    ]);

    const config = new CyclesConfig({
      baseUrl: "http://localhost:7878",
      apiKey: "test-key",
      tenant: "test",
    });
    const client = new CyclesClient(config);

    const response = await client.createReservation({
      idempotency_key: "test-001",
      subject: { tenant: "test" },
      action: { kind: "test", name: "integration" },
      estimate: { unit: "USD_MICROCENTS", amount: 100 },
    });

    expect(response.isSuccess).toBe(true);
    expect(response.getBodyAttribute("reservation_id")).toBe("res-test-001");
  });
});
```

### Testing error handling

```typescript
import { BudgetExceededError, CyclesProtocolError } from "runcycles";

describe("error handling", () => {
  it("creates BudgetExceededError with correct properties", () => {
    const err = new BudgetExceededError("Budget exceeded", {
      status: 409,
      errorCode: "BUDGET_EXCEEDED",
    });

    expect(err.isBudgetExceeded()).toBe(true);
    expect(err.isReservationExpired()).toBe(false);
    expect(err.status).toBe(409);
    expect(err).toBeInstanceOf(CyclesProtocolError);
  });

  it("handles retry-after", () => {
    const err = new CyclesProtocolError("Try again later", {
      status: 409,
      errorCode: "BUDGET_EXCEEDED",
      retryAfterMs: 5000,
    });

    expect(err.retryAfterMs).toBe(5000);
  });
});
```

### Testing withCycles with mocked fetch

Test the full `withCycles` lifecycle by mocking the underlying HTTP calls:

```typescript
import { withCycles, CyclesClient, CyclesConfig, BudgetExceededError } from "runcycles";

describe("withCycles integration", () => {
  afterEach(() => {
    vi.unstubAllGlobals();
  });

  it("executes function within budget lifecycle", async () => {
    mockFetchSequence([
      {
        status: 200,
        body: {
          decision: "ALLOW",
          reservation_id: "r-1",
          affected_scopes: ["tenant:test"],
          expires_at_ms: Date.now() + 60000,
        },
      },
      { status: 200, body: { status: "COMMITTED" } },
    ]);

    const config = new CyclesConfig({ baseUrl: "http://localhost:7878", apiKey: "key", tenant: "test" });
    const client = new CyclesClient(config);

    const guarded = withCycles(
      { estimate: 1000, actionKind: "test", actionName: "unit", client },
      async (input: string) => `Processed: ${input}`,
    );

    const result = await guarded("hello");
    expect(result).toBe("Processed: hello");
  });

  it("throws BudgetExceededError on deny", async () => {
    mockFetchSequence([
      {
        status: 409,
        body: { error: "BUDGET_EXCEEDED", message: "Insufficient balance" },
      },
    ]);

    const config = new CyclesConfig({ baseUrl: "http://localhost:7878", apiKey: "key", tenant: "test" });
    const client = new CyclesClient(config);

    const guarded = withCycles(
      { estimate: 1000, actionKind: "test", actionName: "unit", client },
      async () => "should not run",
    );

    await expect(guarded()).rejects.toThrow(BudgetExceededError);
  });
});
```

### Testing reserveForStream

Mock the client directly to test streaming handle behavior:

```typescript
import { reserveForStream } from "runcycles";
import { CyclesResponse, CyclesConfig } from "runcycles";

describe("reserveForStream", () => {
  it("creates handle with caps", async () => {
    const client = {
      config: new CyclesConfig({ baseUrl: "http://localhost", apiKey: "key" }),
      createReservation: vi.fn().mockResolvedValue(
        CyclesResponse.success(200, {
          decision: "ALLOW",
          reservation_id: "r-stream-1",
          affected_scopes: ["tenant:test"],
          caps: { max_tokens: 4096 },
        }),
      ),
      commitReservation: vi.fn().mockResolvedValue(
        CyclesResponse.success(200, { status: "COMMITTED" }),
      ),
      releaseReservation: vi.fn(),
      extendReservation: vi.fn(),
    };

    const handle = await reserveForStream({
      client: client as any,
      estimate: 5000,
      unit: "USD_MICROCENTS",
      actionKind: "llm.completion",
      actionName: "gpt-4o",
      tenant: "test",
    });

    expect(handle.reservationId).toBe("r-stream-1");
    expect(handle.caps).toEqual({ maxTokens: 4096 });

    // Commit and verify
    await handle.commit(3000, { tokensInput: 100, tokensOutput: 200 });
    expect(client.commitReservation).toHaveBeenCalledOnce();
  });
});
```

## Tips

- **Unit tests**: test business logic without the decorator/annotation/HOF — it has no effect when bypassed
- **Mock CyclesClient**: use Python `MagicMock`, Java `@MockBean`, or TypeScript `vi.fn()` to avoid needing a real server
- **Test both ALLOW and DENY paths**: ensure your code handles budget denial gracefully
- **Test error paths**: verify release is called when functions/methods throw
- **Use HTTP mocking for integration tests**: `pytest-httpx` for Python, Testcontainers for Java, `vi.stubGlobal("fetch")` for TypeScript
- **Python-specific**: `pytest-httpx` mocks both the sync `CyclesClient` and the async `AsyncCyclesClient` (its `httpx_mock` fixture works with async httpx clients too); `respx` is an alternative if you prefer its API
- **TypeScript-specific**: mock `fetch` globally with Vitest's `vi.stubGlobal()` or use `msw` (Mock Service Worker) for more realistic HTTP mocking

## Next steps

- [Error Handling in TypeScript](/how-to/error-handling-patterns-in-typescript) — TypeScript exception handling patterns
- [Error Handling in Python](/how-to/error-handling-patterns-in-python) — Python exception handling patterns
- [Error Handling Patterns](/how-to/error-handling-patterns-in-cycles-client-code) — general error handling patterns
- [Using the Client Programmatically](/how-to/using-the-cycles-client-programmatically) — direct client usage
- [SpEL Expression Reference](/configuration/spel-expression-reference-for-cycles) — expression syntax (Java)



# Troubleshooting and FAQ

Common issues when integrating and operating Cycles, with solutions.

## Reservation and budget issues

### NOT_FOUND on first reservation

**Symptom:** The very first reservation attempt returns `404 NOT_FOUND` with a response message like `"Budget not found for provided scope: tenant:acme-corp"`.

**Cause:** No budget ledger exists for any derived scope in any unit. Creating a tenant does not automatically create a budget. The server checks all scope levels (tenant, workspace, app, etc.) and skips those without a budget — but at least one must exist.

The runtime plane uses a single `NOT_FOUND` wire code for all resource-not-found conditions (missing reservation, missing budget). The `message` field distinguishes them: `"Reservation not found: ..."` vs. `"Budget not found for provided scope: ..."`.

This is distinct from `400 UNIT_MISMATCH` — if a budget exists at the scope but in a different unit (e.g., you requested USD_MICROCENTS but only TOKENS is funded), you get `UNIT_MISMATCH` instead, with `details.expected_units` listing the units that are funded.

On `POST /v1/decide` and `POST /v1/reservations` with `dry_run=true`, the same "no budget" condition doesn't surface as a 404. Those endpoints return `200 OK` with `decision: DENY` and `reason_code: "BUDGET_NOT_FOUND"` in the response body — so a preflight check can distinguish "no budget" from "insufficient budget" without catching an exception.

**Fix:** Create a budget via the admin API:

```bash
curl -s -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "scope": "tenant:acme-corp",
    "unit": "USD_MICROCENTS",
    "allocated": { "amount": 100000000, "unit": "USD_MICROCENTS" }
  }' | jq .
```

Remember: a reservation is checked against every derived scope that has a budget defined. Scopes without budgets are skipped, but at least one derived scope must have a budget. If you have budgets at multiple levels, each one must have sufficient funds.

### BUDGET_EXCEEDED but I just funded the budget

**Symptom:** You funded a budget, but reservations are still denied.

**Possible causes:**

1. **Scope mismatch.** The funded scope does not match the reservation scope. Check that the scope path is exactly right — `tenant:acme-corp` is different from `tenant:acme-corp/workspace:prod`.

2. **Unit mismatch.** You funded in `TOKENS` but the reservation uses `USD_MICROCENTS`. Each unit has its own separate ledger.

3. **Reserved budget.** Other active reservations may be holding budget. Check balances to see the `reserved` field:

```bash
curl -s "http://localhost:7878/v1/balances?tenant=acme-corp" \
  -H "X-Cycles-API-Key: $API_KEY" | jq .
```

The `remaining` field shows available budget after accounting for active reservations.

4. **Hierarchical exhaustion.** A parent scope may be exhausted even if the child scope has budget. Check balances at all levels.

### RESERVATION_EXPIRED — TTL too short

**Symptom:** Commit fails with `410 RESERVATION_EXPIRED` because the LLM call took longer than expected.

**Fixes:**

- **Increase TTL** when creating reservations. The default is 60 seconds (`ttl_ms: 60000`). For long-running operations, use 120 seconds or more.
- **Use automatic heartbeat.** Python `@cycles`, TypeScript `withCycles`, Java `@Cycles`, and Rust `ReservationGuard` schedule from server-authoritative remaining lifetime when available. Ensure you're using a lifecycle helper rather than raw HTTP.
- **For raw HTTP users:** implement the [remaining-lifetime schedule and same-key recovery rules](/protocol/reservation-ttl-grace-period-and-extend-in-cycles), not a blind fixed interval.
- **Recover known spend.** Current lifecycle helpers persist the commit and fall back to a same-key `/v1/events` debit when the reservation has already expired. Low-level callers must persist and perform that recovery themselves.

### DEBT_OUTSTANDING blocking new reservations

**Symptom:** New reservations fail with `409 DEBT_OUTSTANDING` even though the budget was recently funded.

**Cause:** A previous commit with `ALLOW_WITH_OVERDRAFT` created debt. When no `overdraft_limit` is configured (or it is 0), any outstanding debt blocks new reservations until repaid. If an `overdraft_limit > 0` is configured, debt within the limit does not block reservations.

**Fix:** Repay the debt:

```bash
curl -s -X POST "http://localhost:7979/v1/admin/budgets/fund?scope=tenant:acme-corp&unit=USD_MICROCENTS" \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "operation": "REPAY_DEBT",
    "amount": { "amount": 500000, "unit": "USD_MICROCENTS" },
    "idempotency_key": "repay-001"
  }' | jq .
```

### IDEMPOTENCY_MISMATCH on retry

**Symptom:** Retrying a failed request returns `409 IDEMPOTENCY_MISMATCH`.

**Cause:** You're reusing the same idempotency key with a different payload. Idempotency keys must be unique per distinct operation. If the original request *succeeded*, retrying with the same key and same payload returns the original response (safe replay). But if the payload changed, you get a mismatch.

**Fix:** Use a new idempotency key for each distinct operation. Use UUIDs or request-scoped identifiers.

## Authentication and authorization

### UNAUTHORIZED (401)

**Symptom:** All requests fail with `401`.

**Checklist:**

1. Is the `X-Cycles-API-Key` header present in the request?
2. Is the key value correct? (Keys start with `cyc_live_` or `cyc_test_`)
3. Has the key been revoked? Validate it:

```bash
curl -s -X POST http://localhost:7979/v1/auth/validate \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: admin-bootstrap-key" \
  -d '{"key_secret": "cyc_live_..."}' | jq .
```

### FORBIDDEN (403) — tenant mismatch

**Symptom:** Requests return `403 FORBIDDEN`.

**Cause:** The `tenant` field in the reservation subject does not match the tenant associated with the API key.

**Fix:** Ensure the `subject.tenant` matches the API key's tenant. Each API key is scoped to exactly one tenant.

### Missing permissions

**Symptom:** Specific operations fail with `403` even though the API key is valid.

**Cause:** The API key does not have the required permission. Permissions are:

| Operation | Required permission |
|---|---|
| Reserve | `reservations:create` |
| Commit | `reservations:commit` |
| Release | `reservations:release` |
| Extend | `reservations:extend` |
| List reservations | `reservations:list` |
| Balances | `balances:read` |

**Fix:** Create a new API key with the required permissions, or update the existing key's permissions.

## Connection and infrastructure

### Connection refused on port 7878 or 7979

**Symptom:** `ECONNREFUSED` or `Connection refused`.

**Checklist:**

1. Is Docker running? (`docker compose ps`)
2. Are the containers healthy? (`docker compose logs cycles-server`)
3. Is Redis accessible? (`redis-cli -h localhost -p 6379 ping`)
4. Are ports conflicting? Check with `lsof -i :7878` or `netstat -tlnp | grep 7878`.

### Timeout errors

**Symptom:** Requests to Cycles server time out.

**Possible causes:**

1. **Redis is slow or unreachable.** Check Redis connectivity and latency.
2. **Server overloaded.** The reservation Lua scripts are atomic but can queue under very high concurrency.
3. **Network issues.** Ensure the client can reach the server (firewall, DNS, proxy).

**Fix for SDK clients:** Increase the client timeout:

::: code-group
```python [Python]
config = CyclesConfig(
    base_url="http://localhost:7878",
    api_key="cyc_live_...",
    connect_timeout=5.0,  # seconds (default 2.0)
    read_timeout=10.0,    # seconds (default 5.0)
)
```
```typescript [TypeScript]
const config = new CyclesConfig({
  baseUrl: "http://localhost:7878",
  apiKey: "cyc_live_...",
  connectTimeout: 5_000, // ms (default 2000)
  readTimeout: 10_000,   // ms (default 5000)
});
```
:::

## SDK-specific issues

### Python: decorator not working with async functions

**Symptom:** The `@cycles` decorator doesn't seem to work with `async def` functions.

**Fix:** The `@cycles` decorator automatically detects sync vs async functions — no separate decorator is needed. Just use `@cycles` on both:

```python
from runcycles import cycles

# Works with sync functions
@cycles(estimate=5000, action_kind="llm.completion", action_name="gpt-4o")
def ask_sync(prompt: str) -> str:
    ...

# Also works with async functions — auto-detected
@cycles(estimate=5000, action_kind="llm.completion", action_name="gpt-4o")
async def ask_async(prompt: str) -> str:
    ...
```

If you need a fully async programmatic client (not the decorator), use `AsyncCyclesClient`:

```python
from runcycles import AsyncCyclesClient, CyclesConfig

client = AsyncCyclesClient(CyclesConfig.from_env())
```

### TypeScript: streaming response not committing

**Symptom:** Budget is reserved but never committed for streaming calls.

**Cause:** Using `withCycles` for streaming calls. The `withCycles` HOF commits when the wrapped function returns, but streaming functions return before the stream finishes.

**Fix:** Use `reserveForStream` for streaming operations:

```typescript
const handle = await reserveForStream({
  client: cyclesClient,
  estimate: 5000,
  actionKind: "llm.completion",
  actionName: "gpt-4o",
});

try {
  const stream = await openai.chat.completions.create({ stream: true, ... });
  // ... consume stream ...
  await handle.commit(actualCost, { tokensInput, tokensOutput });
} catch (err) {
  await handle.release("stream_error");
  throw err;
}
```

### Spring Boot: @Cycles annotation not intercepting

**Symptom:** Methods annotated with `@Cycles` run without budget enforcement.

**Checklist:**

1. Is `cycles-client-java-spring` on the classpath?
2. Is the `cycles.base-url` property set in `application.yml`?
3. Is the method being called through the Spring proxy? (Direct `this.method()` calls bypass AOP — see below.)
4. Is the class a Spring-managed bean (`@Service`, `@Component`, etc.)?

The most common cause is **self-invocation**: calling a `@Cycles` method from another method in the same class using `this.method()`. Spring's proxy-based AOP cannot intercept these internal calls. The starter logs a `WARN` at startup when it detects beans susceptible to this pattern.

**Fix:** Extract the `@Cycles` method into a separate `@Service`, or self-inject the proxy with `@Lazy @Autowired`. See [Self-Invocation](/quickstart/getting-started-with-the-cycles-spring-boot-starter#self-invocation-internal-method-calls) for full workarounds.

<a id="spring-boot-illegalstateexception-nested-cycles"></a>
### Spring Boot: IllegalStateException — nested @Cycles

**Symptom:** `IllegalStateException("Nested @Cycles not supported")` thrown at runtime.

**Cause:** A `@Cycles`-annotated method called another `@Cycles`-annotated method (even across different beans). The starter prevents this because each reservation is independent — nesting would double-count budget.

**Fix:** Place `@Cycles` at the outermost entry point only. Remove `@Cycles` from inner methods that are called within an already-guarded operation. See [Nesting Prevention](/quickstart/getting-started-with-the-cycles-spring-boot-starter#nesting-prevention) for details.

### TypeScript / Python: nested budget guards double-counting

**Symptom:** Budget is consumed faster than expected when using nested `withCycles` (TypeScript) or `@cycles` (Python) calls.

**Cause:** Unlike Spring, the TypeScript and Python clients do not block nested calls — each guard silently creates an independent reservation. If an outer guard reserves 500 and an inner guard reserves 100, **600 total** is deducted from the budget, not 500.

**Fix:** Place the budget guard at the outermost entry point only. Inner functions should be plain functions without their own guard. See the nesting sections in the [TypeScript](/quickstart/getting-started-with-the-typescript-client#nested-withcycles-calls) and [Python](/quickstart/getting-started-with-the-python-client#nested-cycles-calls) quickstart guides.

## FAQ

### Why can't I delete tenants or budgets?

By design. Cycles uses **status-based lifecycle management** instead of hard deletion for most objects. Tenants, budgets, and reservations are referenced across the system (audit logs, API keys, committed transactions). Deleting them would orphan those records and break audit trails.

Instead, use the cleanup mechanism for each object type:

- **Tenants:** `PATCH status → CLOSED` — blocks all operations, retains data. See [Tenant Lifecycle](/how-to/tenant-creation-and-management-in-cycles#tenant-status-lifecycle).
- **Budgets:** `POST fund` with `RESET` to zero — sets allocated to 0 to block new reservations; retains ledger history. See [Resizing a budget](/how-to/budget-allocation-and-management-in-cycles#resizing-a-budget-reset). (For clearing spent at billing-period boundaries, use `RESET_SPENT` — [Starting a new billing period](/how-to/budget-allocation-and-management-in-cycles#starting-a-new-billing-period-reset-spent).)
- **API Keys:** `DELETE` revokes the key (ACTIVE → REVOKED) but retains the record. See [Revoking API Keys](/how-to/api-key-management-in-cycles#revoking-api-keys).

### Can I use Cycles without Docker?

Yes. Run Redis 7+ natively, build the server JARs with Maven, and start them with `java -jar`. See [Deploy the Full Stack](/quickstart/deploying-the-full-cycles-stack) Option C.

### What happens if the Cycles server goes down?

Your application's behavior depends on your error handling. The SDK clients throw exceptions when the server is unreachable. You should implement a fallback strategy — see [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer).

### Can multiple applications share the same Cycles server?

Yes. Each application uses its own tenant (or its own workspace within a tenant). The Cycles server is stateless — all state lives in Redis.

### How do I prove what Cycles decided, after the fact?

Enable **CyclesEvidence**. When the signing identity is configured, decide / reserve / commit / release responses — and budget/lifecycle denials like a `409 BUDGET_EXCEEDED` — carry an optional `cycles_evidence` reference (`evidence_id` + `cycles_evidence_url`). The `evidence_id` is a SHA-256 content address computed synchronously and returned in-band; the `cycles-server-events` tier asynchronously Ed25519-signs the envelope, which you fetch and verify at `GET /v1/evidence/{id}` — offline, without trusting or reaching the live ledger. It's the receipt, not the gate: it doesn't change enforcement, it makes decisions auditable and bindable by other systems. See [CyclesEvidence: Verifiable Audit](/concepts/cycles-evidence-verifiable-audit-for-agent-decisions), the [envelope reference](/protocol/cycles-evidence-envelopes-in-cycles), and the operator [enablement runbook](https://github.com/runcycles/cycles-server-events/blob/main/docs/evidence-identity-enablement.md).

### What happens to evidence if I don't configure the signing identity?

Nothing breaks in the budget-enforcement path — Cycles continues to decide, reserve, commit, and release exactly as before — but verifiable evidence is not available until the shared identity is configured consistently. In fully unconfigured/dev setups, responses omit `cycles_evidence`; if `EVIDENCE_SERVER_ID` is blank on `cycles-server-events`, the evidence signer is disabled and pending source records are left untouched, not dead-lettered. If `EVIDENCE_SERVER_ID` is present but the producer/worker public identity differs, the worker dead-letters on the `evidence_id` cross-check. The events worker's ephemeral-key mode is development-only and only covers the case where `EVIDENCE_SERVER_ID` is present but the signing pair is absent; it is not a production evidence identity and will not match a runtime server publishing a different public signer. Set `EVIDENCE_SERVER_ID` + `EVIDENCE_SIGNING_SIGNER_DID` (public, on both services) and `EVIDENCE_SIGNING_PRIVATE_KEY_HEX` (secret, on `cycles-server-events` only), then watch `evidence:failed` during rollout.

### How do I reset a budget to zero?

Two different "reset to zero" intents — use the right operation for the one you mean.

**To block new reservations** (allocated → 0, decommission a scope):

```bash
# RESET with amount=0 — sets allocated to 0. Spent is preserved in the ledger
# for historical accounting; remaining goes negative if spent > 0.
curl -s -X POST "http://localhost:7979/v1/admin/budgets/fund?scope=tenant:acme-corp&unit=USD_MICROCENTS" \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{"operation": "RESET", "amount": {"amount": 0, "unit": "USD_MICROCENTS"}, "idempotency_key": "decommission-001"}' | jq .
```

**To start a new billing period** (spent → 0, keep allocated at the new period's ceiling):

```bash
# RESET_SPENT with the new period's allocation — sets allocated AND clears spent.
curl -s -X POST "http://localhost:7979/v1/admin/budgets/fund?scope=tenant:acme-corp&unit=USD_MICROCENTS" \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{"operation": "RESET_SPENT", "amount": {"amount": 1000000000, "unit": "USD_MICROCENTS"}, "idempotency_key": "reset-001", "reason": "Monthly billing period reset"}' | jq .
```

See [Starting a new billing period](/how-to/budget-allocation-and-management-in-cycles#starting-a-new-billing-period-reset-spent) for more detail including the optional `spent` override for migrations, prorated signups, and corrections.

### How do I see what's using my budget?

Check active reservations and balances:

```bash
# Active reservations
curl -s "http://localhost:7878/v1/reservations?tenant=acme-corp&status=ACTIVE" \
  -H "X-Cycles-API-Key: $API_KEY" | jq .

# Balance breakdown
curl -s "http://localhost:7878/v1/balances?tenant=acme-corp" \
  -H "X-Cycles-API-Key: $API_KEY" | jq .
```

### Is there a way to test without a running server?

Use [shadow mode / dry-run](/how-to/shadow-mode-in-cycles-how-to-roll-out-budget-enforcement-without-breaking-production) to evaluate budget policies without enforcing them. For unit tests, mock the `CyclesClient` — see [Testing with Cycles](/how-to/testing-with-cycles).

## MCP server issues

### MCP tool calls not enforcing budget

**Symptom:** The MCP tools respond but reservations are not actually created on your Cycles server.

**Checklist:**

1. Is `CYCLES_API_KEY` set in the MCP server environment? Without it, the server cannot authenticate.
2. Is `CYCLES_BASE_URL` set? **There is no default** — this variable is required. Set it to your Cycles server URL (e.g., `http://localhost:7878` for local development).
3. Is `CYCLES_MOCK` set to `"true"`? Mock mode returns synthetic responses without contacting a real server and performs no live enforcement. Remove it for production use. The MCP server refuses mock mode when `NODE_ENV=production` unless `CYCLES_ALLOW_MOCK_IN_PRODUCTION=true` is also set.
4. Does the API key have the permission for the requested tool? The MCP tool set uses the valid runtime permissions `reservations:create`, `reservations:commit`, `reservations:release`, `reservations:extend`, `reservations:list`, and `balances:read`. There are no separate `decide` or `events:create` permission values.

### MCP server not appearing in Claude Desktop or Cursor

**Symptom:** The agent does not see Cycles tools.

**Checklist:**

1. Is the config file in the right location?
   - Claude Desktop macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`
   - Claude Desktop Windows: `%APPDATA%\Claude\claude_desktop_config.json`
2. Is the JSON valid? A trailing comma or missing brace will silently break the config. Validate with `cat claude_desktop_config.json | jq .`
3. Did you restart the application after editing the config? MCP server configs are read at startup.
4. For Claude Code: did you run `claude mcp add cycles -- npx -y @runcycles/mcp-server`? Check with `claude mcp list`.

### MCP decisions always return ALLOW

**Symptom:** Every reservation or decide call returns `ALLOW` regardless of budget state.

**Cause:** The server is running in mock mode (`CYCLES_MOCK=true`), which returns synthetic `ALLOW` responses. Generated IDs and timestamps are not deterministic.

**Fix:** Remove the `CYCLES_MOCK` environment variable and ensure `CYCLES_BASE_URL` and `CYCLES_API_KEY` are set correctly.

## Admin API issues

### Cannot create budget — 401 on admin API

**Symptom:** `POST /v1/admin/budgets` returns `401 UNAUTHORIZED`.

**Common causes:**

1. **Wrong port.** The admin API runs on port **7979**, not 7878. The protocol API (reservations, commits) runs on 7878.
2. **Wrong header.** Tenant-scoped budget and policy endpoints use `X-Cycles-API-Key` with the required permissions. Admin-only operations such as tenant/key management, audit logs, budget PATCH/freeze/unfreeze, and runtime reservation force-release use `X-Admin-API-Key`. Budget list and fund are dual-auth; admin-key calls must include `tenant_id`.
3. **Missing admin permissions.** Default API keys lack `admin:write`. Create a key with `"permissions": ["admin:read", "admin:write"]` — see [API Key Management](/how-to/api-key-management-in-cycles#available-permissions).

### Budget fund operation has no effect

**Symptom:** You called the fund endpoint but the balance did not change.

**Checklist:**

1. **Scope path mismatch.** The scope in the fund request must exactly match the budget scope. `tenant:acme-corp` is not the same as `tenant:acme-corp/workspace:prod`.
2. **Wrong operation.** The `operation` field must be one of `CREDIT`, `DEBIT`, `RESET`, `RESET_SPENT`, or `REPAY_DEBT`. Common confusion: `RESET` with the same amount as the current allocation is a **no-op by design** — it resizes the allocated ceiling but preserves spent, so `remaining` stays at its current value. If you wanted to clear spent for a new billing period, use `RESET_SPENT`. See [Starting a new billing period](/how-to/budget-allocation-and-management-in-cycles#starting-a-new-billing-period-reset-spent).
3. **Check the response.** The fund endpoint returns the updated balance. Verify the response body confirms the change.

### Fund endpoint returns 404 for workspace budget

**Symptom:** Funding a workspace budget returns `404 NOT_FOUND`.

**Cause:** You may be using the old path-based endpoint format. The fund and patch endpoints accept `scope` and `unit` as **query parameters**, not path variables.

**Fix:** Use query parameters:

```bash
curl -X POST "http://localhost:7979/v1/admin/budgets/fund?scope=tenant:acme/workspace:prod&unit=USD_MICROCENTS" \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "operation": "CREDIT",
    "amount": { "amount": 500000, "unit": "USD_MICROCENTS" }
  }'
```

The same pattern applies to the patch endpoint: `PATCH /v1/admin/budgets?scope=...&unit=...`.

### Tenant creation returns 409

**Symptom:** `POST /v1/admin/tenants` returns `409`.

**Cause:** A tenant with that ID already exists. Tenant IDs are unique. If you are rerunning a setup script, this is expected and safe to ignore.

## Common first-integration mistakes

### Commit fails with 410 RESERVATION_EXPIRED

**Symptom:** Reserve succeeds, but commit returns `410 RESERVATION_EXPIRED`.

**Cause:** The reservation expired before the commit arrived. The default TTL (60000 ms) may be too short for long-running LLM calls. (A `404 NOT_FOUND` on commit is a different problem: the reservation never existed — check the `reservation_id` you are passing.)

**Fix:**

- Increase the `ttl_ms` when creating reservations. For LLM calls, 120000 ms or more is typical.
- Use the SDK decorators (`@cycles` in Python, `withCycles` in TypeScript, `@Cycles` in Spring) which automatically extend TTL via heartbeat.
- For raw HTTP: call `POST /v1/reservations/{id}/extend` periodically before the TTL expires.

### Budget math does not add up

**Symptom:** You funded a budget with `100000000` expecting $100, but it shows as $1.

**Cause:** Cycles uses `USD_MICROCENTS` where **1 dollar = 100,000,000 microcents** (1 microcent = 10⁻⁸ dollars).

Quick reference:

| Amount | USD_MICROCENTS |
|---|---|
| $0.01 (1 cent) | 1,000,000 |
| $1.00 | 100,000,000 |
| $10.00 | 1,000,000,000 |
| $100.00 | 10,000,000,000 |

See [Understanding Units](/protocol/understanding-units-in-cycles-usd-microcents-tokens-credits-and-risk-points) for the full unit reference.

### Scopes not matching — reservation denied despite budget existing

**Symptom:** A budget exists but reservations are still denied with `BUDGET_EXCEEDED`.

**Cause:** The budget scope path does not match any of the reservation's derived scopes. Enforcement checks every derived scope that has a budget defined — scopes without budgets are skipped, but at least one derived scope must have a budget. Common mismatches:

- Budget at `tenant:acme-corp/workspace:prod` but subject uses `workspace=staging`
- Budget at `tenant:acme-corp/workspace:prod` but subject omits `workspace` entirely (the derived scopes are just `tenant:acme-corp`, which has no budget)
- Budget uses a different tenant ID than the one in the subject

**Fix:** Check that the scope path on the budget matches the scopes derived from the reservation subject. Use the [Scope Derivation](/protocol/how-scope-derivation-works-in-cycles) reference to understand which scopes are derived. See also [Tenants, Scopes, and Budgets](/how-to/understanding-tenants-scopes-and-budgets-in-cycles).

## Webhook and event delivery issues

### Webhook endpoint not receiving events

**Symptom:** You created a webhook subscription and events are occurring (e.g., reservations being denied), but your endpoint isn't receiving HTTP requests.

**Checklist:**

1. **Is the events service running?** Webhook delivery requires the events service (`cycles-server-events`) to be deployed and connected to the same Redis instance. Check the management port with `curl http://localhost:9980/actuator/health`.
2. **Is the subscription active?** Subscriptions are auto-disabled after 10 consecutive delivery failures. Check status: `GET /v1/admin/webhooks/{id}` — look for `"status": "DISABLED"`.
3. **Does the subscription match the event type?** Check the `event_types` array in your subscription. If it's empty, the subscription receives all event types. If it lists specific types, the event must match.
4. **Does the scope filter match?** If you set a `scope_filter`, only events whose scope matches the filter will be delivered. See [Scope Filter Syntax](/protocol/webhook-scope-filter-syntax) for matching rules.
5. **Is the endpoint reachable from the events service?** The events service makes outbound HTTP POST requests. Verify network connectivity, DNS resolution, and firewall rules.
6. **Is the endpoint returning 2xx?** Non-2xx responses trigger retries. After 5 retries (exponential backoff: 1s, 2s, 4s, 8s, 16s), the delivery is marked FAILED. After 10 consecutive failures, the subscription is disabled.

**Diagnostic:** Test the subscription manually:
```bash
curl -X POST http://localhost:7979/v1/admin/webhooks/{id}/test \
  -H "X-Admin-API-Key: $ADMIN_KEY"
```
This sends a `system.webhook_test` event to your endpoint. If it arrives, the delivery pipeline works and the issue is in event matching (types or scope filter).

### Events received but signature verification fails

**Symptom:** Your endpoint receives events but HMAC signature verification fails.

**Common causes:**

1. **Wrong signing secret.** The secret must match the one returned when you created the subscription. It cannot be retrieved again — only rotated.
2. **Reading parsed body instead of raw bytes.** Signature is computed over the **raw request body bytes**, not a re-serialized JSON object. Middleware that parses JSON before your verification code runs will produce different bytes.
3. **Encoding mismatch.** The signature is a hex-encoded HMAC-SHA256 digest. Verify you're comparing hex strings, not raw bytes.

**Fix for common frameworks:**
- **Express:** Use `express.raw({ type: 'application/json' })` on the webhook route, not `express.json()`
- **Flask:** Use `request.get_data()`, not `request.json`
- **Spring Boot:** Inject `HttpServletRequest` and read `getInputStream()` before any `@RequestBody` binding

### Subscription auto-disabled

**Symptom:** Events stop arriving. The subscription status is `DISABLED`.

**Cause:** 10 consecutive delivery attempts failed (non-2xx response, timeout, or DNS error). This is a safety mechanism to prevent hammering a broken endpoint.

**Fix:**
1. Fix the underlying endpoint issue (check logs for the failure reason)
2. Re-enable the subscription: `PATCH /v1/admin/webhooks/{id}` with `{"status": "ACTIVE"}`
3. Optionally replay missed events: `POST /v1/admin/webhooks/{id}/replay` with a time range

### Duplicate events received

**Symptom:** Your endpoint processes the same event twice.

**Cause:** Cycles delivers events **at-least-once**. Network timeouts or slow responses can cause the events service to retry a delivery that your endpoint actually processed.

**Fix:** Deduplicate by `event_id`. Track processed event IDs (e.g., in a Redis SET with 24-hour TTL) and skip duplicates:

```python
def handle_webhook(event):
    event_id = event["event_id"]
    if redis.sismember("processed_events", event_id):
        return  # Already processed
    redis.sadd("processed_events", event_id)
    redis.expire("processed_events", 86400)  # 24h TTL
    # Process event...
```

### Events delayed or arriving out of order

**Symptom:** Events arrive minutes after the triggering action, or events from different actions arrive in unexpected order.

**Causes:**

1. **Normal delivery latency.** Events are dispatched asynchronously. Under normal load, latency is sub-second. Under high load or after retries, latency can increase.
2. **Retry backoff.** If your endpoint returned a non-2xx response, the next retry is delayed by exponential backoff (1s → 2s → 4s → 8s → 16s).
3. **Stale delivery protection.** Deliveries older than 24 hours are auto-failed on pickup. If the events service was down for 24+ hours, events from that period are not delivered — use the replay API to recover them.

**Ordering guarantee:** Delivery order is not guaranteed — dispatch uses a shared queue with concurrent consumers, and retried deliveries re-enter out of order. Only the stored event log is ordered; consumers should dedupe and sequence on `event_id` and the envelope `timestamp`.

### Expected event type not firing

**Symptom:** You expect a `budget.threshold_crossed` or `tenant.suspended` event but it never arrives.

**Cause:** Some event types are registered in the protocol before every service emits them. See the [Event Payloads Reference](/protocol/event-payloads-reference) for the current emitted/planned status by category.

**Currently emitted:**
- `reservation.denied`, `reservation.commit_overage`, `reservation.expired`
- `budget.exhausted`, `budget.over_limit_entered`, `budget.debt_incurred`

If you're subscribed to an event type marked as planned, no events will arrive until the emitting service version supports it.

## Debugging production incidents

For deeper incident analysis, the Incident Patterns section documents common production failures with root cause analysis and prevention strategies:

- **[Runaway Agents and Tool Loops](/incidents/runaway-agents-tool-loops-and-budget-overruns-the-incidents-cycles-is-designed-to-prevent)** — agents that loop indefinitely, burning budget on repeated tool calls
- **[Retry Storms](/incidents/retry-storms-and-idempotency-failures)** — retries that double-charge or bypass budget checks
- **[Concurrent Agent Overspend](/incidents/concurrent-agent-overspend)** — race conditions where multiple agents collectively exceed a shared budget
- **[Scope Misconfiguration](/incidents/scope-misconfiguration-and-budget-leaks)** — budget leaks caused by incorrect scope hierarchies

## Next steps

- [Error Codes and Error Handling](/protocol/error-codes-and-error-handling-in-cycles) — complete error code reference
- [Testing with Cycles](/how-to/testing-with-cycles) — testing strategies and fixtures
- [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) — handling budget denial gracefully

## Related concepts

- [Retry storms and idempotency in agent budgets](/blog/retry-storms-and-idempotency-in-agent-budget-systems)
- [Tracking tokens in a streaming LLM response](/blog/tracking-tokens-in-a-streaming-llm-response)
- [Audit trail as a runtime-authority byproduct](/blog/runtime-authority-byproducts-audit-trail-and-attribution-by-default)



# Understanding Tenants, Scopes, and Budgets in Cycles

Cycles enforces budget limits on autonomous execution. To do that, it uses three building blocks that work together:

- **Tenants** — who is spending
- **Scopes** — where enforcement happens in the hierarchy
- **Budgets** — how much is allowed at each scope

Understanding how these three pieces relate is the foundation for designing effective budget governance. This guide explains the model, shows how the pieces connect, and helps you design your own scope structure.

## The three building blocks

<ScopeDiagram />

## Tenants: the isolation boundary

A tenant is the top-level organizational unit in Cycles. It represents an independent entity whose budget is completely isolated from other tenants.

Depending on your platform, a tenant might be:

- a customer in a SaaS product
- an internal department or team
- a partner or reseller
- an environment (production, staging, development)

### How tenant isolation works

Every API key belongs to exactly one tenant. When your application sends a request to the Cycles server, the server derives the **effective tenant** from the API key and enforces that all operations stay within that tenant's boundary:

- Reservations can only be created for the API key's tenant
- Balances can only be queried within the API key's tenant
- Reservations owned by one tenant cannot be accessed by another

This isolation is enforced at the protocol level on every request. A key for tenant A cannot see or modify tenant B's budgets, reservations, or balances — even if someone knows the reservation ID.

For the full tenant lifecycle (creating, listing, updating, suspending, and closing tenants), see [Tenant Creation and Management](/how-to/tenant-creation-and-management-in-cycles).

## Scopes: the budget hierarchy

A scope is a hierarchical path that identifies a specific budget boundary. Scopes are derived from the **Subject** — the set of fields you send with every request that describe who is spending.

### The six standard levels

The Cycles protocol defines a fixed hierarchy of Subject fields:

```
tenant → workspace → app → workflow → agent → toolset
```

When a request includes Subject fields, the server builds scope paths from them in this canonical order. For example, a request with:

```json
{
  "subject": {
    "tenant": "acme-corp",
    "workspace": "prod",
    "app": "chatbot"
  }
}
```

Produces three derived scopes:

1. `tenant:acme-corp`
2. `tenant:acme-corp/workspace:prod`
3. `tenant:acme-corp/workspace:prod/app:chatbot`

Each of these scopes is a separate budget boundary that the server checks.

### Gap-skipping

You do not need to provide all six levels. If you skip a level, the server simply omits it from the scope path.

For example, a request with only `tenant` and `agent`:

```json
{
  "subject": {
    "tenant": "acme-corp",
    "agent": "summarizer-v2"
  }
}
```

Produces two scopes:

1. `tenant:acme-corp`
2. `tenant:acme-corp/agent:summarizer-v2`

The intermediate levels (`workspace`, `app`, `workflow`) are not present and are not checked. This means you only need to create budgets at the levels you actually care about.

### Custom dimensions

For attribution facets that do not fit the standard hierarchy, the Subject supports a `dimensions` field with custom key-value pairs:

```json
{
  "subject": {
    "tenant": "acme-corp",
    "workflow": "support-triage",
    "dimensions": {
      "run": "run-12345",
      "cost_center": "engineering"
    }
  }
}
```

Dimensions are not budget-scope fields — they never derive scopes and servers MAY ignore them for budgeting decisions, but they do serve reporting, enterprise taxonomies, and policy uses (including the v0.1.26 `per_run` action-quota window, which keys off `dimensions.run_id`). To give each execution an enforceable **run budget**, encode the run identifier in a standard Subject field instead (e.g. `workflow: "run-{id}"`, deriving the scope `workflow:run-{id}`) — see [modeling tenant, workflow, and run budgets](/how-to/how-to-model-tenant-workflow-and-run-budgets-in-cycles).

For the full technical specification, see [How Scope Derivation Works](/protocol/how-scope-derivation-works-in-cycles).

## Budgets: the enforcement layer

A budget is an allocation assigned to a specific scope and unit pair. It is the ceiling against which reservations and commits are measured.

### The ledger formula

Each scope tracks a ledger with these fields:

| Field | Meaning |
|---|---|
| `allocated` | Total budget assigned to this scope |
| `spent` | Committed actual usage |
| `reserved` | Currently held by active reservations |
| `remaining` | Available for new reservations |
| `debt` | Negative balance from overdraft commits |

The relationship between these fields:

```
remaining = allocated - spent - reserved - debt
```

A reservation succeeds only if `remaining >= estimate` across all affected scopes.

### Budgets are independent at each scope level

This is a key concept: budgets do **not** automatically propagate between parent and child scopes.

If you set a tenant budget of $100, that does not automatically distribute $100 to child scopes. Each scope where you want enforcement needs its own explicit budget allocation.

For example:

| Scope | Allocated | What it controls |
|---|---|---|
| `tenant:acme-corp` | $100 | Total cap for the tenant |
| `tenant:acme-corp/workspace:prod` | $60 | Cap for production workloads |
| `tenant:acme-corp/workspace:prod/app:chatbot` | $20 | Cap for the chatbot app specifically |

A reservation for the chatbot must pass all three levels — even if the chatbot scope has room, the reservation fails if the tenant or workspace scope is exhausted.

### At least one budget must exist

Scopes without budgets are simply **skipped** during enforcement — a missing budget at one level never causes a denial on its own. You do not need a budget at every possible scope, only at scopes where you want enforcement.

The one requirement: at least one derived scope must have a budget. If **none** of the derived scopes has a budget in the requested unit, the server rejects the request with `404 NOT_FOUND`. That is different from a budget denial — a reservation that exceeds an existing budget is rejected with `409 BUDGET_EXCEEDED`.

For setting up budgets, see [Budget Allocation and Management](/how-to/budget-allocation-and-management-in-cycles).

## How they work together

Here is what happens when a reservation request flows through the system:

```
1. Your app sends a reservation request
   with Subject {tenant: "acme-corp", workspace: "prod", app: "chatbot"}
   and estimate of 500,000 microcents

2. The server derives the API key's tenant → "acme-corp"
   and verifies subject.tenant matches (403 if not)

3. The server derives scopes from the Subject:
   → tenant:acme-corp
   → tenant:acme-corp/workspace:prod
   → tenant:acme-corp/workspace:prod/app:chatbot

4. The server checks budgets at EVERY derived scope that has one,
   atomically (scopes without a budget are skipped):
   ┌─────────────────────────────────────────┬───────────┬────────┐
   │ Scope                                   │ Remaining │ Result │
   ├─────────────────────────────────────────┼───────────┼────────┤
   │ tenant:acme-corp                        │ 5,000,000 │   OK   │
   │ tenant:acme-corp/workspace:prod         │ 3,000,000 │   OK   │
   │ tenant:acme-corp/workspace:prod/app:cb  │   800,000 │   OK   │
   └─────────────────────────────────────────┴───────────┴────────┘

5. ALL scopes pass → reservation is ALLOWED
   Budget is reserved atomically at every scope

6. If ANY scope fails → entire reservation is DENIED
   No partial reservations, no inconsistent state
```

### Allocation flows top-down, pressure flows bottom-up

- **Top-down:** A tenant budget constrains everything beneath it. If the tenant is exhausted, no child scope can reserve budget — even if the child has its own remaining allocation.

- **Bottom-up:** When a child scope runs low, that pressure is visible in balance queries at higher levels. If the chatbot app is consuming most of the workspace budget, you can see that before the workspace itself is exhausted.

## Designing your scope structure

Your scope structure determines where enforcement happens. Start simple and add levels as your needs grow.

### Start with tenant-only

The simplest model: one budget per tenant.

```
tenant:acme-corp → $100
```

Every reservation by this tenant draws from a single pool. This is enough for basic multi-tenant isolation and is the recommended starting point.

### Add workspace for environment separation

Separate production from staging to prevent test runs from consuming production budget:

```
tenant:acme-corp                       → $100  (total cap)
tenant:acme-corp/workspace:prod        → $80   (production cap)
tenant:acme-corp/workspace:staging     → $20   (staging cap)
```

### Add app or workflow for feature-level control

Different features may justify different budgets:

```
tenant:acme-corp                                    → $100
tenant:acme-corp/workspace:prod                     → $80
tenant:acme-corp/workspace:prod/app:chatbot         → $30
tenant:acme-corp/workspace:prod/app:research-agent  → $50
```

### Add per-execution budgets for safety

Use the `workflow` field to cap individual runs (a standard field is required — `dimensions` are not enforceable):

```
tenant:acme-corp                           → $100  (tenant cap)
tenant:acme-corp/workflow:run-xyz-789      → $2    (single run cap)
```

This protects against runaway loops — even if the tenant has plenty of budget, one execution cannot consume more than $2.

### Decision framework

When deciding which scopes to use, ask:

| Question | Scope to add |
|---|---|
| "How much can this customer spend total?" | `tenant` |
| "How much can this environment consume?" | `workspace` |
| "How much can this feature/product use?" | `app` |
| "How much can this type of process consume?" | `workflow` |
| "How much can this individual agent use?" | `agent` |
| "How much can this set of tools cost?" | `toolset` |
| "How much can this single execution consume?" | `workflow: "run-{id}"` (a standard field — `dimensions` are not enforceable) |

You do not need all of them. Most teams start with tenant + one or two additional levels.

For more patterns, see [Common Budget Patterns](/how-to/common-budget-patterns) and [Tenant, Workflow, and Run Budgets](/how-to/how-to-model-tenant-workflow-and-run-budgets-in-cycles).

## Managing scopes in practice

### Creating budgets at each scope level

Budgets are created through the Admin API. You need one budget per scope per unit:

```bash
# Tenant-level budget
curl -s -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "scope": "tenant:acme-corp",
    "unit": "USD_MICROCENTS",
    "allocated": {"amount": 10000000000, "unit": "USD_MICROCENTS"}
  }' | jq .

# Workspace-level budget (within the tenant)
curl -s -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "scope": "tenant:acme-corp/workspace:prod",
    "unit": "USD_MICROCENTS",
    "allocated": {"amount": 8000000000, "unit": "USD_MICROCENTS"}
  }' | jq .

# App-level budget (within the workspace)
curl -s -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "scope": "tenant:acme-corp/workspace:prod/app:chatbot",
    "unit": "USD_MICROCENTS",
    "allocated": {"amount": 3000000000, "unit": "USD_MICROCENTS"}
  }' | jq .
```

### Querying balances across the hierarchy

Check budget state at any level:

```bash
# All balances for a tenant
curl -s "http://localhost:7878/v1/balances?tenant=acme-corp" \
  -H "X-Cycles-API-Key: $API_KEY" | jq .
```

This returns balances at every scope under the tenant, showing `allocated`, `spent`, `reserved`, `remaining`, and `debt` at each level.

### Resetting budgets for billing periods

At the start of a new billing period, reset budgets to their allocation:

```bash
# Reset tenant-level budget
curl -s -X POST "http://localhost:7979/v1/admin/budgets/fund?scope=tenant:acme-corp&unit=USD_MICROCENTS" \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "operation": "RESET_SPENT",
    "amount": {"amount": 10000000000, "unit": "USD_MICROCENTS"},
    "idempotency_key": "reset-april-2026",
    "reason": "Monthly budget reset"
  }' | jq .

# Reset workspace-level budget
curl -s -X POST "http://localhost:7979/v1/admin/budgets/fund?scope=tenant:acme-corp/workspace:prod&unit=USD_MICROCENTS" \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "operation": "RESET_SPENT",
    "amount": {"amount": 8000000000, "unit": "USD_MICROCENTS"},
    "idempotency_key": "reset-ws-prod-april-2026",
    "reason": "Monthly workspace budget reset"
  }' | jq .
```

Reset each scope independently — parent resets do not cascade to children.

### Evolving your scope structure

You can add new scope levels at any time by creating new budget ledgers. Existing reservations are not affected.

To stop enforcing at a scope level, simply stop including that field in your Subject. Budget ledgers without incoming reservations remain idle.

### Scope consistency

The most important practice: **always include the same Subject fields for the same type of request.** If some code paths include `workspace` and others do not, budget enforcement becomes inconsistent — some requests check the workspace scope, others bypass it.

For more on this, see [Scope Misconfiguration and Budget Leaks](/incidents/scope-misconfiguration-and-budget-leaks).

## A complete example

A SaaS platform with two customers, each with production and staging environments, and per-app budgets.

### Step 1: Create tenants

```bash
# Customer A
curl -s -X POST http://localhost:7979/v1/admin/tenants \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{"tenant_id": "customer-a", "name": "Customer A"}'

# Customer B
curl -s -X POST http://localhost:7979/v1/admin/tenants \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{"tenant_id": "customer-b", "name": "Customer B"}'
```

### Step 2: Create API keys

```bash
# API key for Customer A
KEY_A=$(curl -s -X POST http://localhost:7979/v1/admin/api-keys \
  -H "Content-Type: application/json" \
  -H "X-Admin-API-Key: $ADMIN_API_KEY" \
  -d '{
    "tenant_id": "customer-a",
    "name": "prod-key",
    "permissions": ["reservations:create","reservations:commit","reservations:release","balances:read"]
  }' | jq -r '.key_secret')
```

### Step 3: Create budgets at multiple scope levels

```bash
# Customer A: tenant-level cap
curl -s -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "scope": "tenant:customer-a",
    "unit": "USD_MICROCENTS",
    "allocated": {"amount": 10000000000, "unit": "USD_MICROCENTS"}
  }'

# Customer A: production workspace cap
curl -s -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "scope": "tenant:customer-a/workspace:prod",
    "unit": "USD_MICROCENTS",
    "allocated": {"amount": 8000000000, "unit": "USD_MICROCENTS"}
  }'

# Customer A: chatbot app cap within production
curl -s -X POST http://localhost:7979/v1/admin/budgets \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "scope": "tenant:customer-a/workspace:prod/app:chatbot",
    "unit": "USD_MICROCENTS",
    "allocated": {"amount": 3000000000, "unit": "USD_MICROCENTS"}
  }'
```

### Step 4: Make a reservation

```bash
curl -s -X POST http://localhost:7878/v1/reservations \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $KEY_A" \
  -d '{
    "idempotency_key": "example-001",
    "subject": {
      "tenant": "customer-a",
      "workspace": "prod",
      "app": "chatbot"
    },
    "action": {"kind": "llm.completion", "name": "openai:gpt-4o"},
    "estimate": {"amount": 500000, "unit": "USD_MICROCENTS"},
    "ttl_ms": 30000
  }' | jq .
```

The server checks budgets at all three scopes atomically. If all pass, the reservation is allowed. The response includes `affected_scopes` showing which scopes were charged.

### Step 5: Check balances across the hierarchy

```bash
curl -s "http://localhost:7878/v1/balances?tenant=customer-a" \
  -H "X-Cycles-API-Key: $KEY_A" | jq '.balances[] | {scope: .scope_path, remaining: .remaining.amount, reserved: .reserved.amount}'
```

This shows the remaining and reserved amounts at every scope level — giving you visibility into where budget pressure exists in the hierarchy.

## Best practices

### Tenant best practices

- **One tenant per isolation boundary.** If two groups of users should not share budget, they should be separate tenants. Do not multiplex unrelated customers into a single tenant.
- **Use stable, meaningful tenant IDs.** Tenant IDs appear in scope paths, audit logs, and API key bindings. Use domain-meaningful names like `customer-acme` or `dept-engineering`, not internal database IDs. They cannot be changed after creation.
- **Suspend before you close.** Use `SUSPENDED` for temporary blocks (payment failure, investigation). Only use `CLOSED` when you are permanently decommissioning — it is irreversible.
- **Use metadata for external correlation.** Store billing IDs, plan tiers, and external system references in the `metadata` field. This makes it easy to join tenant data with your billing or CRM system.
- **Set `default_commit_overage_policy` at the tenant level.** This establishes a baseline for all scopes. Override per-budget-ledger or per-reservation when specific scopes need different behavior.

### Scope best practices

- **Start with the fewest scope levels that solve your problem.** Tenant-only is a valid starting point. Add workspace, app, or workflow levels only when you need finer control.
- **Keep Subject fields consistent across all code paths.** If some requests include `workspace` and others do not, enforcement becomes inconsistent — some requests bypass the workspace-level check. See [Scope Misconfiguration and Budget Leaks](/incidents/scope-misconfiguration-and-budget-leaks).
- **Use the canonical hierarchy.** The protocol defines `tenant → workspace → app → workflow → agent → toolset`. Map your concepts to these standard levels rather than fighting the ordering.
- **Prefer standard fields over custom dimensions.** Standard fields have built-in scope derivation support. Use `dimensions` only for reporting facets that never need enforcement (e.g., cost centers, regions) — anything you want to budget, including per-run IDs, belongs in a standard field.
- **Validate scope consistency in tests.** Write tests that verify all code paths for the same operation include the same Subject fields. Inconsistencies cause silent budget bypasses.
- **Only create budgets at scopes you need to enforce.** You do not need a budget at every level — scopes without budgets are skipped during enforcement.

### Budget best practices

- **Always create the tenant-level budget first.** The tenant scope is the foundation. Without it, child scope budgets have no parent boundary.
- **Set child scope budgets smaller than parent scope budgets.** A workspace budget of $80 under a tenant budget of $100 makes sense. A workspace budget of $150 under a tenant budget of $100 wastes allocation — the tenant scope will deny before the workspace budget is exhausted.
- **Use idempotency keys on all funding operations.** This prevents double-funding from retries. Use meaningful keys like `fund-acme-march-2026` rather than random UUIDs.
- **Reset budgets at billing period boundaries.** Use the `RESET_SPENT` operation to clear `spent` at period boundaries rather than accumulating `CREDIT` operations. `RESET_SPENT` emits a `budget.reset_spent` event so dashboards can distinguish period boundaries from ceiling changes. Use `RESET` only for resizing the allocated ceiling (plan changes), not for period boundaries — `RESET` preserves spent.
- **Monitor `is_over_limit` and `debt` proactively.** When `debt > 0` and no `overdraft_limit` is configured, new reservations are blocked with `DEBT_OUTSTANDING`. When `debt > overdraft_limit`, the scope enters over-limit state (`OVERDRAFT_LIMIT_EXCEEDED`). Detect these early.
- **`ALLOW_IF_AVAILABLE` is the default overage policy.** It caps charges to available budget and never creates debt. Switch to `REJECT` for hard stops, or `ALLOW_WITH_OVERDRAFT` when exact accounting with debt is needed. Overdraft creates blocking debt that must be explicitly repaid.

## Common questions

### Do I need a budget at every scope level?

No. You only need budgets at scopes where you want enforcement. If you only care about tenant-level caps, create a single budget at `tenant:acme-corp`. Child scopes without budgets are skipped during enforcement.

### What happens if I skip a level in the hierarchy?

If your Subject includes `tenant` and `app` but not `workspace`, the server derives two scopes: `tenant:acme-corp` and `tenant:acme-corp/app:chatbot`. The workspace level is not checked and does not need a budget.

### Can scopes overlap?

No. Each derived scope is an independent budget boundary. The scope `tenant:acme-corp/workspace:prod` is completely separate from `tenant:acme-corp/workspace:staging`. They do not share budget or aggregate balances.

### Can I change my scope structure later?

Yes. Create new budget ledgers at new scopes at any time. Existing budgets and reservations are unaffected. To stop enforcing at a level, simply stop including that field in your Subject.

### How do scopes relate to API keys?

API keys enforce **tenant isolation** — an API key for tenant A cannot operate on tenant B. Scopes enforce **budget hierarchy within a tenant** — different parts of tenant A's organization can have different budget limits.

### Do parent scope budgets automatically include child scope charges?

No. Parent and child scopes are independent ledgers. A reservation at `tenant:acme-corp/workspace:prod/app:chatbot` charges all three scopes independently. The parent scope does not "roll up" child charges — it is charged directly as part of the atomic reservation.

## Next steps

- [Tenant Creation and Management](/how-to/tenant-creation-and-management-in-cycles) — create and manage tenants via the Admin API
- [Budget Allocation and Management](/how-to/budget-allocation-and-management-in-cycles) — fund and adjust budgets at each scope level
- [How Scope Derivation Works](/protocol/how-scope-derivation-works-in-cycles) — the technical protocol reference for scope mechanics
- [Common Budget Patterns](/how-to/common-budget-patterns) — practical recipes for real-world scope hierarchies
- [Tenant, Workflow, and Run Budgets](/how-to/how-to-model-tenant-workflow-and-run-budgets-in-cycles) — multi-level policy design
- [Scope Misconfiguration and Budget Leaks](/incidents/scope-misconfiguration-and-budget-leaks) — what can go wrong with scope design
- [Multi-Tenant AI Cost Control](/blog/multi-tenant-ai-cost-control-per-tenant-budgets-quotas-isolation) — per-tenant budgets, quotas, and isolation for agent platforms



# Upgrading Cycles Safely

Cycles components release independently but share a protocol and Redis state. Treat an upgrade as a fleet change: identify behavior changes, protect Redis, upgrade consumers before producers, and validate one instance before broad rollout.

The [changelog](/changelog) is the current version and release-history authority. The YAML API specifications remain authoritative for wire behavior.

## Current target fleet

| Component | Current version | Compatibility target |
|---|---:|---|
| Runtime specification | `0.1.25.16` document revision | Protocol `v0.1.25` |
| Governance specification | `0.1.25.42` | Governance `v0.1.25` |
| Runtime server | `0.1.25.59` | Runtime specification `0.1.25.16` |
| Admin server | `0.1.25.55` | Governance specification `0.1.25.42` |
| Events service | `0.1.25.25` | Shared v0.1.25 Redis event, delivery, and evidence records |
| Dashboard | `0.1.25.85` | Current admin, runtime, and evidence views |

Client SDK and integration versions are listed in the [current version matrix](/changelog#current-versions). They can be upgraded independently unless their release notes require a newer server feature.

## Before the change window

1. Read every component's release notes between the deployed and target versions.
2. Search for `SECURITY`, `BEHAVIOR CHANGE`, `Migration`, new required environment variables, changed ports, and changed defaults.
3. Confirm the target versions in the [version compatibility table](/changelog#version-compatibility).
4. Run the target fleet against a copy of production-shaped Redis data.
5. Create and validate a complete Redis backup using the [backup and recovery runbook](/how-to/redis-backup-restore-disaster-recovery).
6. Record current image digests, configuration, secrets, Redis version, and queue depths.
7. Confirm rollback images are still available.

Do not use mutable `latest` tags. Pin a version or immutable digest for every service.

## Operator-action matrix

The table below consolidates current v0.1.25 changes that require deployment or client action. Purely additive/internal patches are omitted.

| When crossing | Required action |
|---|---|
| Events service `<0.1.25.9` → `>=0.1.25.9` | Move liveness, readiness, and Prometheus checks from application port `7980` to management port `9980`. Keep both ports internal. |
| Admin server `<0.1.25.24` → `>=0.1.25.24` | Callers that depend on list ordering must pass a supported `sort_by` and `sort_dir`; budget and webhook default ordering changed. |
| Admin server `<0.1.25.28` → `>=0.1.25.28` | Replace new audit queries for `<unauthenticated>` with `__unauth__` or `__admin__`. Historical rows remain queryable until retention expires. |
| Runtime `<0.1.25.45` or admin `<0.1.25.45` → current | Add `X-Admin-API-Key` to aggregate actuator, Prometheus, OpenAPI, and Swagger requests. Liveness/readiness probes remain public. |
| Runtime `<0.1.25.46` → current | Account for default rate limiting on public evidence/JWKS endpoints and handle `429 LIMIT_EXCEEDED` plus `Retry-After`. |
| Runtime `<0.1.25.47` or admin `<0.1.25.35` → current | Expect closed-tenant mutation guards and cascade behavior. Fresh dry-run/decide evaluations deny with `TENANT_CLOSED`; persisting runtime mutations can return `409 TENANT_CLOSED`. |
| Admin `<0.1.25.49` → current | Rewrite webhook bare-prefix `scope_filter` values to the exact or trailing-`*` syntax. “Base plus descendants” needs two subscriptions. |
| Admin `<0.1.25.51` or events `<0.1.25.23` → current | Upgrade all events workers and admin instances to enforce the tenant/admin webhook category boundary across write, dispatch, retry, replay, and last-mile delivery. |
| Admin `<0.1.25.54` or events `<0.1.25.25` → current | Provide the same 32-byte `WEBHOOK_SECRET_ENCRYPTION_KEY`. Missing keys fail startup; plaintext requires the explicit development-only `WEBHOOK_SECRET_ALLOW_PLAINTEXT=true` escape hatch. |
| Events `<0.1.25.25` → current | Review webhook targets against the always-on SSRF baseline. Private, local, metadata, CGNAT, and IPv6 unique-local destinations are denied unless an explicit development policy permits them. |
| Dashboard `<0.1.25.64` → current production stack | Supply the required Redis password and webhook-secret encryption configuration used by the hardened Compose stack; update health checks to readiness endpoints. |

When release notes and this summary disagree, follow the release notes and YAML specifications.

## Rolling order

For additive v0.1.25 patches, upgrade components in this order:

1. **Events service** — it consumes event, delivery, and evidence records produced by the other services.
2. **Runtime and admin servers** — upgrade one instance at a time; either service can precede the other unless a release note names a paired security rollout.
3. **Dashboard** — upgrade after the APIs it calls.
4. **Client SDKs and integrations** — roll out after the server features they consume are available.

Consumer-before-producer reduces the chance that an older worker sees a record introduced by a newer producer. A release-specific order overrides this general sequence.

For paired security fixes, do not declare the rollout complete until every named component and replica is upgraded. For example, the tenant webhook category boundary requires admin `0.1.25.51+` and every events worker at `0.1.25.23+`.

## Upgrade each service

For each component:

1. Remove one instance from traffic or stop one worker.
2. Start the target image with the production configuration.
3. Wait for its readiness endpoint:

   | Service | Readiness |
   |---|---|
   | Runtime | `http://host:7878/actuator/health/readiness` |
   | Admin | `http://host:7979/actuator/health/readiness` |
   | Events | `http://host:9980/actuator/health/readiness` |

4. Inspect startup logs for configuration fallback, encryption, Redis, lease, queue, or schema warnings.
5. Run the component checks below.
6. Keep the instance under observation before replacing the next replica.

Do not run old and new events workers together longer than necessary when a release note says enforcement is per-worker. An old worker can continue processing shared queue items with its old behavior.

## Verification

### Runtime

- Read a representative balance.
- Create and release a small test reservation with a stable idempotency key.
- Repeat the request and verify idempotent replay.
- Verify expected `DENY` behavior against a constrained test scope.
- Check error and latency metrics.

### Admin

- List tenants, budgets, policies, API keys, webhooks, events, and audit records.
- Verify filters, sort order, and cursors used by operator automation.
- Use a test tenant for any mutation probe.
- Confirm operational endpoints require the expected admin header.

### Events

- Verify `dispatch:pending`, `dispatch:processing`, and `dispatch:retry` do not grow unexpectedly.
- Send a webhook test to an approved target.
- Confirm retry, ownership-boundary, SSRF, and auto-disable metrics remain healthy.
- If evidence is enabled, verify pending records become signed envelopes and the public JWKS resolves the signer.

### Dashboard

- Log in through the production reverse proxy.
- Load overview, tenants, budgets, reservations, webhooks, events, audit, and evidence views.
- Confirm the login/sidebar shows the target dashboard version and no backend API-shape errors appear.

## Rollback

Before rolling back:

1. stop the rollout and block new writes if state compatibility is uncertain;
2. read the release notes for storage migrations or one-way behavior changes;
3. preserve logs and the current Redis state;
4. roll back one instance in isolation and validate it against a copy of the current Redis dataset.

Current v0.1.x wire changes are designed to be additive, and the service processes are stateless. That does not make every behavior rollback invisible: a newer service may already have written records, changed webhook subscriptions, closed tenants, emitted events, or consumed queue items that an older service will not undo.

Do not restore Redis merely to roll back application binaries. A Redis restore is a data-loss operation relative to every write after the restore point and requires the full [disaster-recovery procedure](/how-to/redis-backup-restore-disaster-recovery).

## Post-upgrade record

Record:

- deployed versions and immutable image digests;
- start/end time and operator;
- migration actions completed;
- readiness and smoke-test results;
- queue-depth and error-rate comparison;
- rollback point and backup checksum;
- any deferred client or integration upgrades.

## Related

- [Changelog and Current Versions](/changelog)
- [Deploying the Full Cycles Stack](/quickstart/deploying-the-full-cycles-stack)
- [Production Operations](/how-to/production-operations-guide)
- [Redis Backup, Restore, and Disaster Recovery](/how-to/redis-backup-restore-disaster-recovery)
- [Server Configuration Reference](/configuration/server-configuration-reference-for-cycles)



# Using Bulk Actions for Tenants, Webhooks, and Budgets

Bulk actions let a single admin call suspend hundreds of tenants, pause a fleet of noisy webhooks, reactivate a batch after an incident is resolved, or roll every budget to a new billing period. They ship in `cycles-server-admin` v0.1.25.26 (tenants + webhooks) and v0.1.25.29 (budgets), against governance spec v0.1.25.21 and .26 respectively, and surface in the [Cycles Admin Dashboard](/quickstart/deploying-the-cycles-dashboard) as filter-then-bulk lanes on the Tenants, Webhooks, and Budgets pages.

The endpoints:

| Endpoint | Supported actions | Since |
|----------|-------------------|-------|
| `POST /v1/admin/tenants/bulk-action` | `SUSPEND`, `REACTIVATE`, `CLOSE` | v0.1.25.26 |
| `POST /v1/admin/webhooks/bulk-action` | `PAUSE`, `RESUME`, `DELETE` | v0.1.25.26 |
| `POST /v1/admin/budgets/bulk-action` | `CREDIT`, `DEBIT`, `RESET`, `REPAY_DEBT`, `RESET_SPENT` | v0.1.25.29 |

All three accept the same envelope and return the same response shape. Budget bulk-action has two extra requirements covered in the [Budget bulk-action](#budget-bulk-action) section below.

## Request shape

Bulk actions operate on a filter expression, not an explicit ID list. You describe the target population with the same filters the list endpoints accept, then the server matches and applies the action atomically per row.

```bash
curl -X POST http://localhost:7979/v1/admin/tenants/bulk-action \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "SUSPEND",
    "idempotency_key": "ops-2026-04-17-freeze-abusers",
    "expected_count": 42,
    "filter": {
      "status": "ACTIVE",
      "search": "trial-"
    }
  }'
```

### Required fields

- **`action`** — one of the action values supported by the endpoint (see table above). Unknown values return `400 INVALID_REQUEST`.
- **`idempotency_key`** — stable, unique string. Replays within 15 minutes return the original response without re-executing. Required on every bulk call — there is no "best-effort" mode.
- **`filter`** — an object with the same filter keys the corresponding list endpoint supports. Tenant filters: `status`, `parent_tenant_id`, `observe_mode`, `search`. Webhook filters: `tenant_id`, `status`, `event_type`, `search`. An empty filter is rejected — the server refuses to act on "every tenant" or "every webhook" without at least one constraint. Unknown filter keys return `400 INVALID_REQUEST` (strict `additionalProperties: false`).

`observe_mode` is a forward-compatible tenant filter for the v0.1.26 action-governance preview. v0.1.25.x reference admin servers accept it in the filter shape but only servers that implement observe mode narrow tenant bulk actions by that value.

### Optional fields

- **`expected_count`** — safety gate. If the server resolves the filter to a different number of rows, the call fails with `409 COUNT_MISMATCH` and **no rows are touched**. Use this to catch drift between when you previewed the list and when you executed the bulk action.

## Response envelope

```json
{
  "action": "SUSPEND",
  "idempotency_key": "ops-2026-04-17-freeze-abusers",
  "total_matched": 42,
  "succeeded": [
    { "id": "tenant-abc" },
    { "id": "tenant-def" }
  ],
  "failed": [
    {
      "id": "tenant-ghi",
      "error_code": "INVALID_TRANSITION",
      "message": "cannot SUSPEND from CLOSED"
    }
  ],
  "skipped": [
    {
      "id": "tenant-jkl",
      "reason": "ALREADY_IN_TARGET_STATE"
    }
  ]
}
```

Every row ends in exactly one of the three buckets:

- **`succeeded`** — the row transitioned to the target state. Rows carry `id` only.
- **`failed`** — the row matched the filter but the action could not apply (typically `INVALID_TRANSITION` — e.g., resuming a `DISABLED` webhook, suspending a `CLOSED` tenant). Rows carry `error_code` and `message`.
- **`skipped`** — the row matched the filter but was already in the target state (e.g., a tenant already suspended when `action=SUSPEND`, a webhook already paused). Rows carry `reason` (`ALREADY_IN_TARGET_STATE` or `ALREADY_DELETED`). Not an error — the bulk action is idempotent per row.

`total_matched` equals `succeeded.length + failed.length + skipped.length`. If you supplied `expected_count`, they are guaranteed equal — otherwise the call returned `409 COUNT_MISMATCH` before any row executed.

## Safety gates

### 500-row ceiling — `LIMIT_EXCEEDED`

Bulk actions cap at **500 matched rows per call**. If your filter resolves to more than 500 rows, the server returns HTTP 400 with `error: LIMIT_EXCEEDED`:

```json
{
  "error": "LIMIT_EXCEEDED",
  "message": "filter matches more than 500 tenants; narrow the filter and retry",
  "request_id": "req_...",
  "details": { "total_matched": 637 }
}
```

`total_matched` is the exact server-counted result size, including when it exceeds the 500-row execution cap. No rows are touched. To proceed, narrow the filter (add `status`, `search`, or a scoping field) and run multiple calls with distinct idempotency keys.

### Count mismatch — `COUNT_MISMATCH`

If `expected_count` is provided and disagrees with the resolved match, the call returns HTTP 409:

```json
{
  "error": "COUNT_MISMATCH",
  "message": "expected_count 42 differs from server-counted matches 40",
  "request_id": "req_...",
  "details": { "total_matched": 40 }
}
```

Again, no rows are touched. Re-preview the list and retry with a corrected `expected_count`, or drop the gate if you accept the drift.

### Replay semantics

Bulk calls are idempotent on `idempotency_key`. A replay within the 15-minute window returns the original response verbatim — the server does not re-evaluate the filter on replay. After the window expires, the same key re-executes from scratch against live data.

## Audit trail

One audit entry is written per bulk invocation (not per row). As of v0.1.25.30 its metadata captures the full per-row outcome plus filter echo and wall-clock duration — enough to triage a failure without re-running the op or capturing the synchronous response:

```json
{
  "operation": "bulkActionTenants",
  "resource_type": "tenant",
  "resource_id": "bulk-action",
  "status": 200,
  "metadata": {
    "action": "SUSPEND",
    "total_matched": 42,
    "succeeded": 40,
    "failed": 1,
    "skipped": 1,
    "succeeded_ids": ["tenant_1", "tenant_2", "..."],
    "failed_rows": [
      {"id": "tenant_7", "error_code": "INVALID_TRANSITION", "message": "Already SUSPENDED"}
    ],
    "skipped_rows": [
      {"id": "tenant_9", "reason": "ALREADY_IN_TARGET_STATE"}
    ],
    "filter": { "status": "ACTIVE", "search": "trial-" },
    "duration_ms": 1245,
    "idempotency_key": "ops-2026-04-17-freeze-abusers"
  }
}
```

Worst-case audit row size is ~40 KB at the 500-row bulk cap. Audit tooling that caps on entry-level JSON size should review.

Query bulk-action entries:

```bash
curl -G "http://localhost:7979/v1/admin/audit/logs" \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  --data-urlencode "operation=bulkActionTenants,bulkActionWebhooks,bulkActionBudgets" | jq .
```

The `operation` param was promoted to an array in v0.1.25.27 — you can OR across all three bulk operations in one query.
There is no `bulk_idempotency_key` audit query parameter. The bulk request's `idempotency_key` is stored as `metadata.idempotency_key` on the returned audit row, so filter by `operation` plus `tenant_id`, time range, `trace_id`, or `request_id`, then inspect the expanded row or exported JSON.

## Event log emission

Bulk actions also emit first-class Events on every successful row — one Event per mutated object, matching the kinds the single-op paths emit. Shipped in server versions:

| Endpoint | Per-row Event since | Spec |
|---|---|---|
| `POST /v1/admin/tenants/bulk-action` | admin v0.1.25.38 | v0.1.25.32 |
| `POST /v1/admin/budgets/bulk-action` | admin v0.1.25.38 | v0.1.25.32 |
| `POST /v1/admin/webhooks/bulk-action` | admin v0.1.25.39 | v0.1.25.33 |

The event kinds are the same ones the single-op endpoints emit — `tenant.suspended`, `tenant.reactivated`, `tenant.closed` for the tenant path; `budget.funded`, `budget.debited`, `budget.reset`, `budget.reset_spent`, `budget.debt_repaid` for the budget path; `webhook.paused`, `webhook.resumed`, `webhook.deleted` for the webhook path (see [Event Payloads Reference](/protocol/event-payloads-reference#webhook-lifecycle-events)).

### Correlation IDs

Every per-row emit from one bulk invocation shares a single `correlation_id`:

| Endpoint | Correlation ID shape |
|---|---|
| Tenants | `tenant_bulk_action:<action>:<request_id>` |
| Budgets | `budget_bulk_action:<action>:<request_id>` |
| Webhooks | `webhook_bulk_action:<action>:<request_id>` |

`<request_id>` is the `X-Request-Id` header the client supplied, or `req_<uuid>` when the header was absent (admin v0.1.25.40 replaced the earlier `"no-req"` literal so concurrent header-less invocations don't collide on one correlation_id). To pull every Event a single bulk invocation produced, query `GET /v1/admin/events?correlation_id=<value>`.

### CLOSE is the two-axis case

For `action=CLOSE` on tenants, each mutated row yields **two** correlation axes:

- The parent `tenant.closed` Event carries `correlation_id = tenant_bulk_action:close:<request_id>` — one value shared across every closed tenant in the invocation. Query by this ID to reconstruct *the invocation*.
- Each tenant's cascade fan-out (budgets closed, webhooks disabled, API keys revoked, reservations released — see [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics)) carries `correlation_id = tenant_close_cascade:<tenant_id>:<request_id>`. Query by this ID to reconstruct *one tenant's close*.

The two axes are independent and both are present in the event log. Use whichever matches the question you're answering.

### What does not emit

- **Skipped rows** (`ALREADY_IN_TARGET_STATE` — the row was already in the target status) emit no Event. Matches single-op behavior: a no-op doesn't write to the Event log.
- **Failed rows** (`INVALID_TRANSITION`, etc.) emit no Event. The bulk-action response's `failed[]` bucket and the aggregate `AuditLogEntry` are the operator-facing signals for failures; duplicating to the Event log would produce false failure alerts on any consumer pattern-matching on event kinds.
- **Event emission failures** are caught and logged at WARN; they never abort the bulk op or revert the row's state transition.

## Budget bulk-action

Budget bulk-action (v0.1.25.29) follows the same envelope as tenants and webhooks with two differences: `filter.tenant_id` is REQUIRED, and most actions require an `amount`.

```bash
# End-of-month period rollover for one tenant
curl -X POST http://localhost:7979/v1/admin/budgets/bulk-action \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "filter": { "tenant_id": "acme-corp", "unit": "USD_MICROCENTS" },
    "action": "RESET_SPENT",
    "amount": { "amount": 1000000, "unit": "USD_MICROCENTS" },
    "expected_count": 8,
    "idempotency_key": "period-rollover-2026-05-01-acme"
  }'

# Debt cleanup on over-limit budgets
curl -X POST http://localhost:7979/v1/admin/budgets/bulk-action \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "filter": { "tenant_id": "acme-corp", "has_debt": true },
    "action": "REPAY_DEBT",
    "amount": { "amount": 500000, "unit": "USD_MICROCENTS" },
    "expected_count": 3,
    "idempotency_key": "debt-cleanup-2026-04-18-acme"
  }'
```

### Differences vs. tenants / webhooks

- **`filter.tenant_id` is REQUIRED.** Cross-tenant budget bulk is explicitly out of scope — returns 400 if blank. If you're operating across many tenants, iterate over tenants and make one bulk call per tenant.
- **`amount` is required for all 5 actions.** `CREDIT`, `DEBIT`, `RESET`, `RESET_SPENT`, `REPAY_DEBT` all move value; there is no "state transition only" action like `SUSPEND` on tenants.
- **`spent` is honored only on `RESET_SPENT`.** Use it to override the post-reset `spent` value (for prorated signups, migrations, or credit-back). Default is 0.
- **Optional filters:** `scope_prefix`, `unit`, `status`, `over_limit`, `has_debt`, `utilization_min`, `utilization_max`, `search`. Same shape as `listBudgets`.
- **Per-row idempotency.** The server derives `{idempotency_key}:{scope}:{unit}` per row and passes it to the underlying fund path, so retrying the failed subset on a tighter filter cannot double-apply CREDIT / DEBIT / RESET / RESET_SPENT / REPAY_DEBT against rows that already landed.
- **Per-row `error_code`:** `BUDGET_EXCEEDED` (DEBIT would take remaining negative), `INVALID_TRANSITION` (unit mismatch / FROZEN / CLOSED), `NOT_FOUND` (ledger deleted between match and apply), `INTERNAL_ERROR`.
- **Per-row `skipped` reasons.** Today only `REPAY_DEBT` on `debt==0` produces `ALREADY_IN_TARGET_STATE`.

### When RESET_SPENT vs. RESET

`RESET` resizes the `allocated` ceiling and preserves `spent`, `reserved`, and `debt`. Use it for plan changes ("this tenant upgraded from 500k to 1M").

`RESET_SPENT` clears (or overrides) `spent` and preserves `allocated`, `reserved`, and `debt`. Use it for billing-period rollovers where outstanding reservations and debt must survive the boundary. See [Rolling Over Billing Periods with RESET_SPENT](/how-to/rolling-over-billing-periods-with-reset-spent).

## Recommended pattern

1. **Preview.** Call the matching list endpoint (`GET /v1/admin/tenants` or `GET /v1/admin/webhooks`) with the same filter. Note `total_count` if the server returns it, or paginate to count manually.
2. **Propose.** Compose the bulk request body. Set `idempotency_key` to something traceable back to an incident or runbook (`ops-INC-842-suspend-abusers`). Set `expected_count` to the preview count.
3. **Execute.** POST the bulk request. Capture the full response envelope to your runbook record.
4. **Reconcile.** Inspect `failed[]`. Investigate each `error_code` — bulk actions do not "retry until green"; follow-up fixes are manual.
5. **Audit.** Query audit logs by bulk `operation` and relevant top-level filters, then inspect `metadata.idempotency_key`, `succeeded_ids`, `failed_rows`, and `skipped_rows` to confirm every row was logged and to export for compliance review.

::: tip Dashboard equivalent
The Tenants and Webhooks pages in the [dashboard](/quickstart/deploying-the-cycles-dashboard) expose the same flow as a visual lane: filter the list, preview the count, click **Bulk action**, confirm with a blast-radius summary, and see per-row results in a side panel. The dashboard sets `expected_count` automatically from the current filter count.
:::

## Error reference

| HTTP | `error` | Meaning |
|------|---------|---------|
| 400 | `LIMIT_EXCEEDED` | Filter matched more than 500 rows. Narrow the filter. |
| 400 | `INVALID_REQUEST` | Unknown `action`, empty `filter`, unknown filter key (strict `additionalProperties: false`), or missing `idempotency_key`. |
| 401 | `UNAUTHORIZED` | Invalid or missing `X-Admin-API-Key`. |
| 409 | `COUNT_MISMATCH` | `expected_count` disagreed with resolved match count. Re-preview. |
| Per-row (`error_code` in `failed[]`) | `INVALID_TRANSITION`, `NOT_FOUND`, `PERMISSION_DENIED`, `BUDGET_EXCEEDED`, `INTERNAL_ERROR` | Row-level failure — HTTP status is still 200. |
| Per-row (`reason` in `skipped[]`) | `ALREADY_IN_TARGET_STATE`, `ALREADY_DELETED` | Row-level no-op skip — HTTP status is still 200. |

## Next steps

- [Tenant Creation and Management](/how-to/tenant-creation-and-management-in-cycles) — the single-entity endpoints bulk actions are built on
- [Managing Webhooks](/how-to/managing-webhooks) — per-subscription operations
- [Admin API reference](/admin-api/) — full OpenAPI
- [Searching and Sorting Admin List Endpoints](/how-to/searching-and-sorting-admin-list-endpoints) — how to narrow the filter before a bulk call



# Using the Cycles Client Programmatically

The decorator / annotation handles most use cases automatically. But sometimes you need direct control — building requests manually, managing the lifecycle yourself, or calling endpoints that the decorator does not cover.

The Python `CyclesClient`, Java `CyclesClient`, and TypeScript `CyclesClient` provide the core runtime operations: decide; reserve, commit, release, and extend; reservation list/get; balances; and usage events. They do not expose helpers for every public or preview endpoint, such as evidence retrieval and JWKS discovery; use direct HTTP for an endpoint your client's current API does not cover.

::: warning Low-level clients own settlement durability
Programmatic reserve/commit calls do not give the SDK lifecycle helper enough context to persist an application-owned settlement before its first request. Once your operation knows its actual usage, durably store the exact commit or event body and idempotency key before sending it. Reuse that key after ambiguous outcomes, and switch an expired commit to `POST /v1/events`. The Python decorator/stream helper, TypeScript `withCycles`/stream handle, Spring `@Cycles`, and async Rust `ReservationGuard` implement this recovery profile automatically.
:::

## Getting the client

In Java (Spring Boot Starter), `CyclesClient` is auto-configured and available for injection.

::: code-group
```python [Python]
from runcycles import CyclesClient, CyclesConfig

config = CyclesConfig(
    base_url="http://localhost:7878",
    api_key="cyc_live_...",
    tenant="acme-corp",
)

client = CyclesClient(config)
```
```java [Java]
@Service
public class BudgetService {

    private final CyclesClient cyclesClient;

    public BudgetService(CyclesClient cyclesClient) {
        this.cyclesClient = cyclesClient;
    }
}
```
```typescript [TypeScript]
import { CyclesClient, CyclesConfig } from "runcycles";

const config = new CyclesConfig({
  baseUrl: "http://localhost:7878",
  apiKey: "cyc_live_...",
  tenant: "acme-corp",
});

const client = new CyclesClient(config);
```
:::

Or from environment variables (Python and TypeScript):

::: code-group
```python [Python]
config = CyclesConfig.from_env()  # reads CYCLES_BASE_URL, CYCLES_API_KEY, etc.
client = CyclesClient(config)
```
```typescript [TypeScript]
const config = CyclesConfig.fromEnv(); // reads CYCLES_BASE_URL, CYCLES_API_KEY, etc.
const client = new CyclesClient(config);
```
:::

::: tip TypeScript naming convention
`CyclesClient` methods accept and return wire-format (snake_case) JSON. For camelCase convenience, use the typed mapper functions (`reservationCreateRequestToWire`, `reservationCreateResponseFromWire`, etc.) documented in the [TypeScript Getting Started guide](/quickstart/getting-started-with-the-typescript-client#programmatic-client).
:::

## Creating a reservation

::: code-group
```python [Python]
from runcycles import (
    CyclesClient, ReservationCreateRequest,
    Subject, Action, Amount, Unit,
)

with CyclesClient(config) as client:
    response = client.create_reservation(ReservationCreateRequest(
        idempotency_key="req-abc-123",
        subject=Subject(tenant="acme", workspace="production", app="chatbot"),
        action=Action(kind="llm.completion", name="gpt-4o"),
        estimate=Amount(unit=Unit.USD_MICROCENTS, amount=5000),
        ttl_ms=60_000,
    ))

    if not response.is_success:
        raise RuntimeError(f"Reservation failed: {response.error_message}")

    reservation_id = response.get_body_attribute("reservation_id")
    decision = response.get_body_attribute("decision")

    # For non-dry-run reservations, insufficient budget returns 409 (not decision=DENY).
    # decision=DENY in a 2xx response only occurs when dry_run=true.

    # Proceed with work...
```
```java [Java]
ReservationCreateRequest request = ReservationCreateRequest.builder()
    .idempotencyKey(UUID.randomUUID().toString())
    .subject(Subject.builder()
        .tenant("acme")
        .workspace("production")
        .app("chatbot")
        .build())
    .action(new Action("llm.completion", "gpt-4o", null))
    .estimate(new Amount(Unit.USD_MICROCENTS, 5000L))
    .ttlMs(60000L)
    .build();

CyclesResponse<Map<String, Object>> response = cyclesClient.createReservation(request);

if (!response.is2xx()) {
    throw new RuntimeException("Reservation failed: " + response.getErrorMessage());
}

Map<String, Object> body = response.getBody();
String reservationId = (String) body.get("reservation_id");
String decision = (String) body.get("decision");

// For non-dry-run reservations, insufficient budget returns 409 (not decision=DENY).
// decision=DENY in a 2xx response only occurs when dry_run=true.

// Proceed with work...
```
```typescript [TypeScript]
import { CyclesClient, CyclesConfig, Unit } from "runcycles";

const response = await client.createReservation({
  idempotency_key: "req-abc-123",
  subject: { tenant: "acme", workspace: "production", app: "chatbot" },
  action: { kind: "llm.completion", name: "gpt-4o" },
  estimate: { unit: Unit.USD_MICROCENTS, amount: 5000 },
  ttl_ms: 60_000,
});

if (!response.isSuccess) {
  throw new Error(`Reservation failed: ${response.errorMessage}`);
}

const reservationId = response.getBodyAttribute("reservation_id") as string;
const decision = response.getBodyAttribute("decision") as string;

// For non-dry-run reservations, insufficient budget returns 409 (not decision=DENY).
// decision=DENY in a 2xx response only occurs when dry_run=true.

// Proceed with work...
```
:::

## Committing actual usage

::: code-group
```python [Python]
from runcycles import CommitRequest, CyclesMetrics

client.commit_reservation(reservation_id, CommitRequest(
    idempotency_key="commit-abc-123",
    actual=Amount(unit=Unit.USD_MICROCENTS, amount=3200),
    metrics=CyclesMetrics(
        tokens_input=150,
        tokens_output=80,
        latency_ms=320,
        model_version="gpt-4o-2024-08-06",
    ),
    metadata={"app_request_id": "req-abc-123"},
))
```
```java [Java]
CyclesMetrics metrics = new CyclesMetrics();
metrics.setTokensInput(150);
metrics.setTokensOutput(80);
metrics.setLatencyMs(320);
metrics.setModelVersion("gpt-4o-2024-08-06");

CommitRequest commitRequest = CommitRequest.builder()
    .idempotencyKey("commit-" + UUID.randomUUID())
    .actual(new Amount(Unit.USD_MICROCENTS, 3200L))
    .metrics(metrics)
    .metadata(Map.of("app_request_id", "req-abc-123"))
    .build();

CyclesResponse<Map<String, Object>> commitResponse =
    cyclesClient.commitReservation(reservationId, commitRequest);
```
```typescript [TypeScript]
await client.commitReservation(reservationId, {
  idempotency_key: "commit-abc-123",
  actual: { unit: Unit.USD_MICROCENTS, amount: 3200 },
  metrics: {
    tokens_input: 150,
    tokens_output: 80,
    latency_ms: 320,
    model_version: "gpt-4o-2024-08-06",
  },
  metadata: { app_request_id: "req-abc-123" },
});
```
:::

## Releasing a reservation

If work is cancelled or fails before producing any usage:

::: code-group
```python [Python]
from runcycles import ReleaseRequest

client.release_reservation(reservation_id, ReleaseRequest(
    idempotency_key="release-abc-123",
    reason="Task cancelled by user",
))
```
```java [Java]
ReleaseRequest releaseRequest = ReleaseRequest.builder()
    .idempotencyKey("release-" + UUID.randomUUID())
    .reason("Task cancelled by user")
    .build();

cyclesClient.releaseReservation(reservationId, releaseRequest);
```
```typescript [TypeScript]
await client.releaseReservation(reservationId, {
  idempotency_key: "release-abc-123",
  reason: "Task cancelled by user",
});
```
:::

## Full lifecycle example

::: code-group
```python [Python]
from runcycles import (
    CyclesClient, CyclesConfig, ReservationCreateRequest, CommitRequest,
    ReleaseRequest, Subject, Action, Amount, Unit, CyclesMetrics,
)

config = CyclesConfig(base_url="http://localhost:7878", api_key="cyc_live_...", tenant="acme")

def process_document(doc_id: str, content: str) -> str:
    idempotency_key = f"doc-{doc_id}"
    estimated_tokens = len(content) // 4

    with CyclesClient(config) as client:
        # 1. Reserve
        response = client.create_reservation(ReservationCreateRequest(
            idempotency_key=idempotency_key,
            subject=Subject(tenant="acme", workspace="production", app="doc-processor"),
            action=Action(kind="llm.completion", name="gpt-4o"),
            estimate=Amount(unit=Unit.USD_MICROCENTS, amount=estimated_tokens * 10),
            ttl_ms=120_000,
            overage_policy="ALLOW_IF_AVAILABLE",
        ))

        if not response.is_success:
            raise RuntimeError(f"Reservation failed: {response.error_message}")

        reservation_id = response.get_body_attribute("reservation_id")

        # 2. Execute
        try:
            result = call_llm(content)

            # 3. Commit
            actual_tokens = count_tokens(result)
            client.commit_reservation(reservation_id, CommitRequest(
                idempotency_key=f"commit-{idempotency_key}",
                actual=Amount(unit=Unit.USD_MICROCENTS, amount=actual_tokens * 10),
                metrics=CyclesMetrics(
                    tokens_input=estimated_tokens,
                    tokens_output=actual_tokens,
                ),
            ))
            return result

        except Exception:
            # 4. Release on failure
            client.release_reservation(reservation_id, ReleaseRequest(
                idempotency_key=f"release-{idempotency_key}",
                reason="Processing failed",
            ))
            raise
```
```java [Java]
@Service
public class DocumentProcessor {

    private final CyclesClient cyclesClient;

    public DocumentProcessor(CyclesClient cyclesClient) {
        this.cyclesClient = cyclesClient;
    }

    public String processDocument(String docId, String content) {
        String idempotencyKey = "doc-" + docId;
        int estimatedTokens = content.length() / 4;

        // 1. Reserve
        ReservationCreateRequest reservation = ReservationCreateRequest.builder()
            .idempotencyKey(idempotencyKey)
            .subject(Subject.builder()
                .tenant("acme")
                .workspace("production")
                .app("doc-processor")
                .build())
            .action(new Action("llm.completion", "gpt-4o", null))
            .estimate(new Amount(Unit.USD_MICROCENTS, (long) estimatedTokens * 10))
            .ttlMs(120000L)
            .overagePolicy(CommitOveragePolicy.ALLOW_IF_AVAILABLE)
            .build();

        CyclesResponse<Map<String, Object>> reserveResponse =
            cyclesClient.createReservation(reservation);

        if (!reserveResponse.is2xx()) {
            throw new CyclesProtocolException("Reservation failed: "
                + reserveResponse.getErrorMessage());
        }

        String reservationId = (String) reserveResponse.getBody().get("reservation_id");
        // For non-dry-run reservations, a 2xx response means decision is ALLOW or ALLOW_WITH_CAPS.
        // Insufficient budget returns 409 (handled above by !is2xx check).

        // 2. Execute
        try {
            String result = callLlm(content);

            // 3. Commit
            int actualTokens = countTokens(result);
            CyclesMetrics commitMetrics = new CyclesMetrics();
            commitMetrics.setTokensInput(estimatedTokens);
            commitMetrics.setTokensOutput(actualTokens);

            CommitRequest commit = CommitRequest.builder()
                .idempotencyKey("commit-" + idempotencyKey)
                .actual(new Amount(Unit.USD_MICROCENTS, (long) actualTokens * 10))
                .metrics(commitMetrics)
                .build();

            cyclesClient.commitReservation(reservationId, commit);
            return result;

        } catch (Exception e) {
            // 4. Release on failure
            cyclesClient.releaseReservation(reservationId,
                ReleaseRequest.builder()
                    .idempotencyKey("release-" + idempotencyKey)
                    .reason("Processing failed: " + e.getMessage())
                    .build());
            throw e;
        }
    }
}
```
```typescript [TypeScript]
import { CyclesClient, CyclesConfig, Unit } from "runcycles";

const config = new CyclesConfig({
  baseUrl: "http://localhost:7878",
  apiKey: "cyc_live_...",
  tenant: "acme",
});

async function processDocument(docId: string, content: string): Promise<string> {
  const idempotencyKey = `doc-${docId}`;
  const estimatedTokens = Math.ceil(content.length / 4);
  const client = new CyclesClient(config);

  // 1. Reserve
  const response = await client.createReservation({
    idempotency_key: idempotencyKey,
    subject: { tenant: "acme", workspace: "production", app: "doc-processor" },
    action: { kind: "llm.completion", name: "gpt-4o" },
    estimate: { unit: Unit.USD_MICROCENTS, amount: estimatedTokens * 10 },
    ttl_ms: 120_000,
    overage_policy: "ALLOW_IF_AVAILABLE",
  });

  if (!response.isSuccess) {
    throw new Error(`Reservation failed: ${response.errorMessage}`);
  }

  const reservationId = response.getBodyAttribute("reservation_id") as string;

  // 2. Execute
  try {
    const result = await callLlm(content);

    // 3. Commit
    const actualTokens = countTokens(result);
    await client.commitReservation(reservationId, {
      idempotency_key: `commit-${idempotencyKey}`,
      actual: { unit: Unit.USD_MICROCENTS, amount: actualTokens * 10 },
      metrics: {
        tokens_input: estimatedTokens,
        tokens_output: actualTokens,
      },
    });
    return result;
  } catch (err) {
    // 4. Release on failure
    await client.releaseReservation(reservationId, {
      idempotency_key: `release-${idempotencyKey}`,
      reason: "Processing failed",
    });
    throw err;
  }
}
```
:::

## Preflight decision check

Check budget availability without creating a reservation.

::: code-group
```python [Python]
from runcycles import DecisionRequest

response = client.decide(DecisionRequest(
    idempotency_key="decide-001",
    subject=Subject(tenant="acme", workspace="production"),
    action=Action(kind="llm.completion", name="gpt-4o"),
    estimate=Amount(unit=Unit.USD_MICROCENTS, amount=50_000),
))

decision = response.get_body_attribute("decision")  # "ALLOW", "ALLOW_WITH_CAPS", or "DENY"
if decision == "DENY":
    print("Budget low — show warning in UI")
```
```java [Java]
DecisionRequest decisionRequest = DecisionRequest.builder()
    .idempotencyKey("decide-" + UUID.randomUUID())
    .subject(Subject.builder()
        .tenant("acme")
        .workspace("production")
        .build())
    .action(new Action("llm.completion", "gpt-4o", null))
    .estimate(new Amount(Unit.USD_MICROCENTS, 50000L))
    .build();

CyclesResponse<Map<String, Object>> decisionResponse = cyclesClient.decide(decisionRequest);
String decision = (String) decisionResponse.getBody().get("decision");

if ("DENY".equals(decision)) {
    // Show "budget low" warning in UI
}
```
```typescript [TypeScript]
const decisionResponse = await client.decide({
  idempotency_key: "decide-001",
  subject: { tenant: "acme", workspace: "production" },
  action: { kind: "llm.completion", name: "gpt-4o" },
  estimate: { unit: Unit.USD_MICROCENTS, amount: 50_000 },
});

const decision = decisionResponse.getBodyAttribute("decision") as string;
if (decision === "DENY") {
  console.log("Budget low — show warning in UI");
}
```
:::

## Querying balances

::: code-group
```python [Python]
response = client.get_balances(tenant="acme", workspace="production")
if response.is_success:
    for balance in response.body.get("balances", []):
        # remaining is a SignedAmount object: {"unit": ..., "amount": ...}
        remaining = balance["remaining"]
        print(f"Scope: {balance['scope']}, remaining: {remaining['amount']} {remaining['unit']}")
```
```java [Java]
Map<String, String> params = Map.of(
    "tenant", "acme",
    "workspace", "production"
);

CyclesResponse<Map<String, Object>> balanceResponse = cyclesClient.getBalances(params);

// Balance amounts are objects ({unit, amount}), not raw numbers. Use the
// typed BalanceQueryResult / Balance accessors instead of casting to Number.
BalanceQueryResult result = BalanceQueryResult.fromMap(balanceResponse.getBody());

for (Balance balance : result.getBalances()) {
    SignedAmount remaining = balance.getRemaining();  // can be negative (overdraft)
    Amount spent = balance.getSpent();
    Amount reserved = balance.getReserved();
    System.out.printf("Scope: %s, remaining: %d %s, spent: %d, reserved: %d%n",
        balance.getScope(),
        remaining.getAmount(), remaining.getUnit(),
        spent.getAmount(), reserved.getAmount());
}
```
```typescript [TypeScript]
const balanceResponse = await client.getBalances({ tenant: "acme", workspace: "production" });
if (balanceResponse.isSuccess) {
  const balances = balanceResponse.getBodyAttribute("balances") as Array<Record<string, unknown>>;
  for (const balance of balances ?? []) {
    console.log(`Scope: ${balance.scope}, remaining: ${JSON.stringify(balance.remaining)}`);
  }
}
```
:::

## Listing reservations

::: code-group
```python [Python]
response = client.list_reservations(tenant="acme", status="ACTIVE", limit="20")
if response.is_success:
    for reservation in response.body.get("reservations", []):
        print(f"ID: {reservation['reservation_id']}, status: {reservation['status']}")
```
```java [Java]
Map<String, String> params = Map.of(
    "tenant", "acme",
    "status", "ACTIVE",
    "limit", "20"
);

CyclesResponse<Map<String, Object>> listResponse =
    cyclesClient.listReservations(params);
```
```typescript [TypeScript]
const listResponse = await client.listReservations({
  tenant: "acme",
  status: "ACTIVE",
  limit: "20",
});
if (listResponse.isSuccess) {
  const reservations = listResponse.getBodyAttribute("reservations") as Array<Record<string, unknown>>;
  for (const r of reservations ?? []) {
    console.log(`ID: ${r.reservation_id}, status: ${r.status}`);
  }
}
```
:::

## Recording events (direct debit)

For post-hoc accounting without a reservation.

::: code-group
```python [Python]
from runcycles import EventCreateRequest

response = client.create_event(EventCreateRequest(
    idempotency_key="evt-001",
    subject=Subject(tenant="acme", workspace="production"),
    action=Action(kind="search.api", name="google-search"),
    actual=Amount(unit=Unit.USD_MICROCENTS, amount=1200),
))
```
```java [Java]
EventCreateRequest event = EventCreateRequest.builder()
    .idempotencyKey("evt-" + UUID.randomUUID())
    .subject(Subject.builder()
        .tenant("acme")
        .workspace("production")
        .build())
    .action(new Action("search.api", "google-search", null))
    .actual(new Amount(Unit.USD_MICROCENTS, 1200L))
    .build();

cyclesClient.createEvent(event);
```
```typescript [TypeScript]
await client.createEvent({
  idempotency_key: "evt-001",
  subject: { tenant: "acme", workspace: "production" },
  action: { kind: "search.api", name: "google-search" },
  actual: { unit: Unit.USD_MICROCENTS, amount: 1200 },
});
```
:::

## CyclesResponse

All client methods return a `CyclesResponse` (in Java, `CyclesResponse<Map<String, Object>>`):

::: code-group
```python [Python]
response = client.create_reservation(request)

response.is_success          # True if HTTP 2xx
response.is_server_error     # True if HTTP 5xx
response.is_transport_error  # True if connection failed
response.status              # HTTP status code
response.body                # Parsed JSON body as dict
response.error_message       # Error message (if error)
response.request_id          # X-Request-Id header
response.rate_limit_remaining  # X-RateLimit-Remaining (int or None)
```
```java [Java]
CyclesResponse<Map<String, Object>> response = cyclesClient.createReservation(request);

response.is2xx();           // true if HTTP 2xx
response.is5xx();           // true if HTTP 5xx
response.isTransportError();// true if connection failed
response.getStatus();       // HTTP status code
response.getBody();         // parsed JSON body as Map
response.getErrorMessage(); // error message (if error)
```
```typescript [TypeScript]
const response = await client.createReservation(request);

response.isSuccess;          // true if HTTP 2xx
response.isServerError;      // true if HTTP 5xx
response.isTransportError;   // true if connection failed
response.status;             // HTTP status code
response.body;               // Parsed JSON body (wire format)
response.errorMessage;       // Error message (if error)
response.requestId;          // X-Request-Id header
response.rateLimitRemaining; // X-RateLimit-Remaining (number or undefined)
response.cyclesTenant;       // X-Cycles-Tenant header
```
:::

## Async support (Python)

The Python client provides `AsyncCyclesClient` for asyncio-based applications:

```python
from runcycles import AsyncCyclesClient

async with AsyncCyclesClient(config) as client:
    response = await client.create_reservation(request)
    if response.is_success:
        reservation_id = response.get_body_attribute("reservation_id")
        # ... do async work ...
        await client.commit_reservation(reservation_id, commit_request)
```

## When to use programmatic vs decorator/annotation

| Use case | Approach |
|---|---|
| Wrapping a single method call in a budget lifecycle | `@cycles` decorator / `@Cycles` annotation / `withCycles` HOF |
| Managing multiple reservations in a workflow | Programmatic `CyclesClient` |
| Querying balances or listing reservations | Programmatic `CyclesClient` |
| Preflight decisions for UI routing | Programmatic `CyclesClient` |
| Recording events without reservations | Programmatic `CyclesClient` |
| Fine-grained error handling per step | Programmatic `CyclesClient` |

## Next steps

- [Getting Started with the TypeScript Client](/quickstart/getting-started-with-the-typescript-client) — TypeScript HOF and streaming adapter setup
- [Getting Started with the Python Client](/quickstart/getting-started-with-the-python-client) — Python decorator and client setup
- [Getting Started with the Spring Boot Starter](/quickstart/getting-started-with-the-cycles-spring-boot-starter) — Java annotation-based approach
- [API Reference](/api/) — interactive endpoint documentation
- [Error Handling in TypeScript](/how-to/error-handling-patterns-in-typescript) — TypeScript exception hierarchy and patterns
- [Error Handling in Python](/how-to/error-handling-patterns-in-python) — Python exception hierarchy and patterns
- [Error Handling Patterns](/how-to/error-handling-patterns-in-cycles-client-code) — general error handling patterns
- [SDK Settlement Recovery and Durability](/protocol/sdk-settlement-recovery-and-durability) — durable lifecycle-helper guarantees and the low-level-client boundary



# Using the Cycles Dashboard

The [Cycles Admin Dashboard](/quickstart/deploying-the-cycles-dashboard) is a Vue 3 SPA that sits in front of `cycles-server-admin` and `cycles-server`. Everything it does is a call against those two backends — the dashboard itself holds no state. This page is the operator's tour: how to log in, what every page does, and which features are behind which admin key capability.

If you haven't deployed the dashboard yet, start with [Deploy the Cycles Admin Dashboard](/quickstart/deploying-the-cycles-dashboard). The examples below assume the dashboard is reachable at `https://admin.example.com`.

## Login and capability gating

The only credential the dashboard accepts is an admin API key. On the login page:

1. Enter the admin API key (the value of `ADMIN_API_KEY` on the server).
2. The dashboard calls `GET /v1/auth/introspect` to validate the key and retrieve the capability set.
3. Sidebar navigation, action buttons, and page access are all gated by capability booleans returned by introspect (`view_overview`, `view_budgets`, `manage_budgets`, `manage_reservations`, etc.). The `manage_*` flags default to "allow" when a server doesn't return them — only an explicit `false` hides the corresponding actions.

The key is stored in `sessionStorage` — it survives a page refresh but is cleared when the tab closes. It is never written to `localStorage` or a cookie. Idle timeout is 30 minutes; absolute timeout is 8 hours; the check runs every 15 seconds.

After 3 failed login attempts the dashboard enforces exponential backoff (5s → 10s → 20s → 40s → 60s cap). A 401 from any subsequent API call clears the session and redirects to login (with one carve-out: a 401 caused by calling an endpoint the running admin server doesn't have yet is surfaced as an in-view error instead of a logout). A 403 — authenticated key, forbidden operation — keeps the session and surfaces the error in the view, and network failures or timeouts never end the session.

::: tip Treat the admin key like a root credential
There is no user login, no SSO out of the box. Rotate the key regularly, keep it in a secrets manager, and consider putting the dashboard behind SSO or VPN. The dashboard does not weaken this — it uses whatever key you give it.
:::

## The eleven views

| View | Purpose |
|------|---------|
| Overview | Aggregated health — counter strip, four donut charts (budget status / utilization / events by category / webhook fleet), and six attention cards for actionable work. See [Overview screen](#overview-screen). |
| Tenants | Tenant list with parent/child hierarchy columns and bulk actions |
| Tenant detail (`/tenants/:id`) | Per-tenant drill-down with nested Budgets / API Keys / Policies tabs, spend rollup, children list, and a parent-tenant breadcrumb |
| Budgets | Tenant-scoped budget list with utilization and debt bars; inline `RESET` and `RESET_SPENT` |
| Events | Correlation-first investigation tool with expandable detail rows |
| API Keys (`/api-keys`) | Cross-tenant key list with masked IDs, permissions, status filters |
| Webhooks | Subscription health (green / yellow / red) with status filters and bulk actions |
| Webhook detail (`/webhooks/:id`) | Four-stat row (last-success chip, delivery-outcome donut, attempts histogram, response-time p50/p95/max — see [WebhookDetailView stats row](#webhookdetailview-stats-row-v0-1-25-51)), delivery history, last error, signature rotation, pause/resume, replay, and test |
| Reservations (`/reservations`) | Hung-reservation force-release during incident response (runtime-plane admin-on-behalf-of) |
| Audit | Compliance query tool with CSV / JSON export |
| Evidence (`/evidence`) | Signed evidence-envelope viewer — paste a 64-hex `evidence_id` (or follow the deep link from force-release) to retrieve the envelope and check the signer key against the published JWK Set |

Most pages poll their backends on a page-specific interval — see the [deployment guide](/quickstart/deploying-the-cycles-dashboard#polling-cadence) for the cadence table. Audit is manual-only: you press **Run Query** explicitly to avoid drive-by queries against retention-expensive endpoints.

## Overview screen

The Overview is the landing page. It opens on eight parallel fetches — the `/v1/admin/overview` aggregate plus seven list queries (API keys, the last 10 audit entries, budgets at ≥ 90% utilization, frozen budgets, closed tenants, budgets with debt, and webhooks) — that together hydrate the counter strip, the four donut charts, the six attention cards, and the recent-operator-activity feed. The fetches resolve independently, so one flaky endpoint degrades to an error banner instead of blanking the page.

The payload includes optional aggregates beyond the visible cards. `recent_denials_by_reason` is populated by v0.1.25.x admin servers and lets operators see denial distribution even when the recent-event sample is capped. `quota_health`, `access_control_stats`, and `tenant_counts.in_observe_mode` are reserved for v0.1.26+ action-governance servers; dashboards should render them when present and tolerate null or absent values on v0.1.25.x reference servers. See [Action Governance Preview](/protocol/action-governance-preview-in-cycles).

### Counter strip

Top of page. Four tiles — Tenants, Budgets, Webhooks, and Events — each showing a server-aggregated total plus status chips (e.g. active / frozen / over / debt) that drill to the corresponding list view with the filter pre-applied. The Events tile's time window is server-driven: the overview payload carries `event_window_seconds`, and the tile renders it as "Events (Xm)" — there is no client-side window selector. Counter totals come from the server aggregate, so they reconcile by construction with the list pages' own counts (no client-side reduce drift).

### The four donuts (v0.1.25.47–.52)

Beneath the counter strip sits a 4-up donut grid. Every slice is clickable and drills to the corresponding filtered list view — chart and list read from the same server aggregate so the numbers match.

| Donut | Slices | Slice-click target |
|---|---|---|
| Budget status | Active / Frozen / Over-limit / Closed | `/budgets?status=ACTIVE\|FROZEN\|CLOSED` or `/budgets?filter=over_limit` |
| Budget utilization | Healthy (<90%) / Near cap (90–99%) / Over cap (≥100%) | `/budgets?utilization_min=…&utilization_max=…` (integer percent, v0.1.25.50) |
| Events by category | `budget` / `reservation` / `tenant` / `api_key` / `policy` / `webhook` / `system` / `runtime` | `/events?category=<name>&from=<window-start>&to=<now>` — time window mirrors the counter-strip "Events (Xm)" window (v0.1.25.53) |
| Webhook fleet health | Active / Paused / Disabled | `/webhooks?status=ACTIVE\|PAUSED\|DISABLED` |

Each card title carries a muted "· click a slice" hint to telegraph interactivity. Dark-mode palette re-derives on toggle (the charts aren't just re-skinned images — they're vue-echarts instances driven by a reactive `useChartTheme` composable). Spec-terminal CLOSED budgets are filtered out of the utilization bucketing and total (v0.1.25.59) so a CLOSED budget at 120% doesn't inflate "Over cap" and CLOSED budgets don't inflate "Healthy" — FROZEN stays included because it's non-terminal. Independently, the client-fetched attention cards (every card except Recent denials, which comes from the server aggregate) exclude rows owned by CLOSED tenants (v0.1.25.45) so the transient Mode-B cascade window doesn't surface un-actionable work. Screen readers get an auto-rendered `sr-only` data table per pie chart (v0.1.25.56).

Under the donuts, six attention cards surface actionable work: Budgets at or near cap, Budgets with debt, Frozen budgets, Failing webhooks, Expiring API keys (7d), and Recent denials (1h). Each card's "View all" link carries the same filter the card applied, so drill-down and card count agree by construction. An alert banner above the counter strip enumerates whichever cards are firing as severity-colored jump-link pills.

## Power-user features

### Command palette — `Cmd+K` / `Ctrl+K`

Press `Cmd+K` on macOS or `Ctrl+K` on Linux/Windows to open the palette. It is a navigation tool, not an action runner — it never mutates anything and applies no capability gating. Two modes:

- **Tenant fuzzy search (default).** Type a tenant name or ID fragment; the palette filters a cached tenant list (60s TTL, up to 150 prefetched with a "Load more" affordance) and Enter jumps to the tenant detail page.
- **Slash commands.** Type `/` to list them: `/wh <subscription_id>` (or `/webhook`) opens a webhook detail page, `/tenant <tenant_id>` (or `/t`) opens a tenant by exact ID, `/key <key_id>` opens the Audit view filtered by that key, `/audit <log_id or resource_id>` searches the audit log, and `/event <event_id>` filters the Events view. Budget and reservation ID jumps are intentionally not offered — those views don't honor the needed URL filters yet.

### Bulk action lanes

The Tenants, Webhooks, and Budgets pages expose a filter-then-bulk workflow:

1. Apply filters in the page toolbar (`status`, `plan`, `over_limit`, etc.) until the row count is what you want to act on.
2. Click **Bulk action**. A side panel opens with the `expected_count` pre-filled from the current filter.
3. Pick the action. Tenants: `SUSPEND`, `REACTIVATE`, `CLOSE`. Webhooks: `PAUSE`, `RESUME`, `DELETE`. Budgets (v0.1.25.35+, requires admin v0.1.25.29+): `CREDIT`, `DEBIT`, `RESET`, `RESET_SPENT`, `REPAY_DEBT`. A blast-radius summary confirms before execution.
4. The dashboard calls `POST /v1/admin/tenants/bulk-action`, `/v1/admin/webhooks/bulk-action`, or `/v1/admin/budgets/bulk-action` with the filter, the `expected_count` safety gate, and an idempotency key generated from the current session.
5. The result panel shows per-row `succeeded`, `failed`, `skipped` lists — rendered in a `BulkActionResultDialog` (v0.1.25.34+) with per-row copy-ID affordances and operator-friendly error messages sourced from the shared `errorCodeMessages` catalog. Failed rows show the per-row `error_code`.

**Row-select variant (v0.1.25.36).** The Budgets view also supports row-select bulk Freeze and Unfreeze — select individual checkboxes across filtered rows rather than applying to the whole filter. This is a client-side fan-out over the per-budget freeze/unfreeze endpoints, not a `POST /v1/admin/budgets/bulk-action` request. Row-select bulk failures open the same `BulkActionResultDialog` with per-row status.

See [Using Bulk Actions](/how-to/using-bulk-actions-for-tenants-and-webhooks) for the full request shape and error taxonomy.

### Cross-surface correlation chip (v0.1.25.39)

Rows on the Events and Audit views and event-timeline entries carry **correlation chips** for up to three identifiers — `trace_id`, `request_id`, `correlation_id` (see [Correlation and Tracing](/protocol/correlation-and-tracing-in-cycles) for what each one scopes). Webhook delivery-history rows do not render the chip affordance; the delivery **export** includes `trace_id` for offline joins. Clicking a chip pivots to the other view with that identifier pre-applied as a filter:

- Click `trace_id` on an Audit row → EventsView filtered to the same trace.
- Click `trace_id` on an Events row → AuditView filtered to the originating entry.
- Click `correlation_id` on an EventTimeline row → EventsView filtered to all events in the same cluster (v0.1.25.37+).
- Copy-to-clipboard icon on the chip for sharing into tickets or chat.

There is no pivot menu or deliveries side panel — the chip is a filtered navigation plus a copy affordance. Operator triage in v0.1.25 starts here: pull a `trace_id` out of a failing response header (`X-Cycles-Trace-Id`) or error body, paste it into the Audit page's `trace_id` filter (or the Events view), and follow the chips between surfaces. Requires `cycles-server-admin` v0.1.25.31+ for server-side support. See [Correlation and Tracing](/protocol/correlation-and-tracing-in-cycles).

### Terminal-state row toggle (v0.1.25.46)

List defaults are endpoint-specific: Tenants and API Keys use `created_at desc`, Budgets use `utilization desc`, and Webhooks inherit the admin default `consecutive_failures desc`. Terminal rows can still crowd operational views or displace the active rows operators need most, so v0.1.25.46 added an explicit visibility toggle.

Tenants, Budgets, Webhooks, and API Keys now hide terminal rows by default and surface a "Show closed (N)" / "Show disabled (N)" / "Show revoked (N)" toggle with the hidden count. Flipping the toggle partitions the list so active rows stay on top and terminal rows drop to the bottom — column-sort order is preserved within each group. Matches the GitHub / Linear / Gmail convention for done / archived items.

| View | Terminal definition |
|---|---|
| Tenants | `status=CLOSED` |
| Budgets | `status=CLOSED` (FROZEN stays visible — it's non-terminal) |
| Webhooks | `status=DISABLED` |
| API Keys | `status IN (REVOKED, EXPIRED)` |

Toggle state mirrors to URL as `?include_terminal=1` on the top-level Tenants, Budgets, and Webhooks views so deep-links survive across reloads. The top-level API Keys view and tenant-detail sub-tabs do not mirror the toggle to the URL. Picking a terminal status explicitly from the dropdown (for example, `status=CLOSED`) auto-engages the toggle so the list isn't silently empty.

### WebhookDetailView stats row (v0.1.25.51)

Clicking a webhook subscription opens `/webhooks/:id`. Between the subscription card and the Delivery History table sits a four-up stat row that aggregates over recently-loaded deliveries:

| Stat | Meaning |
|---|---|
| Last success | Chip with traffic-light semantics — green if < 1h, amber 1h–24h, red ≥ 24h or no successful delivery on file. The fastest visual check that a subscription is still delivering. |
| Delivery outcome | Donut partitioning loaded deliveries by status (success / retrying / failed / stale). Clicking a slice sets the delivery-table status filter in place — no route change, because the filter is local. |
| Attempts per delivery | Histogram bucketed 0 / 1 / 2 / 3 / 4 / 5+ with a severity color ramp. Makes retry storms visible before you scan rows. |
| Response time | p50 / p95 / max computed via NIST nearest-rank over deliveries that carry `response_time_ms`. |

The stats aggregate whatever deliveries the history table has loaded — there's no second fetch. Scroll / Load More on the table re-computes the stats in place.

### Freshness pill on page headers (v0.1.25.54)

Polling list views show a small muted "Updated Xm ago" pill on the `PageHeader`, beside the refresh button. It reads `usePolling.lastSuccessAt` — successful polls update it; failed polls leave it alone, so operators can tell at a glance whether they're looking at fresh data or a silent poll outage. Absent on manual-query pages (Audit) and on views that don't poll.

### Parent-tenant breadcrumb

Tenant detail pages show a **Parent** link when the tenant has a `parent_tenant_id`, and a Children list of sub-tenants. Clicking a child threads `?parent=<source>` into the URL so the back arrow returns to the tenant you came from (single hop — deeper A → B → C chains return to the immediately-previous tenant, not the root). There is no scope breadcrumb: the dashboard does not render a `tenant → workspace → app` trail; scope paths appear only as budget/reservation row data.

### RESET_SPENT inline funding

On the Budgets page, every row has a funding dropdown. Alongside `CREDIT`, `DEBIT`, `REPAY_DEBT`, and `RESET`, the dropdown exposes `RESET_SPENT` — the v0.1.25.18+ funding operation that sets `allocated` to the supplied amount and resets `spent` (to zero, or an explicit override) while preserving `reserved` and `debt`. Picking it opens a confirmation dialog where you enter the new allocation and can either leave `spent` at zero (monthly rollover) or enter an explicit starting value (prorated correction).

See [Rolling Over Billing Periods with RESET_SPENT](/how-to/rolling-over-billing-periods-with-reset-spent) for when to use each pattern.

### Closed-tenant tombstone and cascade preview

As of v0.1.25.43 (consuming admin v0.1.25.36), the dashboard surfaces tenant-close cascade behavior through four coordinated affordances:

- **Closed-tenant banner.** When `tenant.status === 'CLOSED'`, an amber read-only banner renders at the top of `TenantDetailView`: *"Tenant closed — all owned objects are read-only."* Immediately answers the "why won't this unfreeze?" question on closed-tenant pages.
- **CLOSE confirm-dialog cascade preview.** Before closing, the confirmation dialog enumerates what will be terminated — owned budgets, webhook subscriptions, API keys, open reservations, with counts pulled from already-loaded tenant-detail state. Spells out *"This cannot be undone."* Useful for estimating blast radius before pulling the trigger.
- **`TENANT_CLOSED` 409 humanizer.** Any mutation that races the cascade (stale tab, deep-link, in-flight request) surfaces as *"Tenant is closed — this object is read-only."* instead of a raw 409. Lives alongside the existing error-code map in `errorCodeMessages.ts`.
- **Tenant-cascade audit + event chip.** `AuditView` and `EventTimeline` rows render a small amber "tenant cascade" chip when the event carries a `_via_tenant_cascade` suffix (`budget.closed_via_tenant_cascade`, `webhook.disabled_via_tenant_cascade`, `api_key.revoked_via_tenant_cascade`, `reservation.released_via_tenant_cascade`, or audit operation `tenant_close_cascade`). Lets operators visually distinguish cascade-triggered state changes from user-driven ones when correlating by `correlation_id`.

Requires admin v0.1.25.36. Running the dashboard against admin `.32` still renders the tombstone + dialog preview (pure client-side), but the cascade itself won't fire and frozen budgets on closed tenants continue to inflate the Overview alert counter. Running against `.35` works for the common cascade path (budgets + reservations are cascaded and their mutations return `TENANT_CLOSED`), but policy / api-key / webhook-admin mutations against closed-tenant objects still go through without the Rule 2 guard — `.36` closes those remaining endpoints.

See [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics) for the full protocol contract.

## Incident-response actions

Every destructive action is one-click with a confirmation and a blast-radius summary:

| Action | Page | Backend call |
|--------|------|-------------|
| Freeze budget | Budgets / Budget detail | `POST /v1/admin/budgets/freeze?scope={scope}&unit={unit}` |
| Unfreeze budget | Budgets | `POST /v1/admin/budgets/unfreeze?scope={scope}&unit={unit}` |
| Bulk Freeze / Unfreeze budgets (v0.1.25.36+) | Budgets — row-select + floating toolbar | Client-side fan-out over `POST /v1/admin/budgets/freeze` / `unfreeze` |
| Suspend tenant | Tenants / Tenant detail | `PATCH /v1/admin/tenants/{id}` |
| Reactivate tenant | Tenants | `PATCH /v1/admin/tenants/{id}` |
| Revoke API key | API Keys | `DELETE /v1/admin/api-keys/{id}` |
| Pause webhook | Webhooks / Webhook detail | `PATCH /v1/admin/webhooks/{id}` |
| Resume webhook | Webhooks | `PATCH /v1/admin/webhooks/{id}` |
| Test webhook | Webhook detail | `POST /v1/admin/webhooks/{id}/test` |
| Replay webhook events | Webhook detail | `POST /v1/admin/webhooks/{id}/replay` |
| Force-release reservation | Reservations / Reservation detail | `POST /v1/reservations/{id}/release` with `X-Admin-API-Key` |
| Emergency tenant-wide freeze | Tenant detail | Bulk freeze across all budgets for the tenant |
| Close tenant (cascades owned objects, v0.1.25.43+) | Tenants / Tenant detail | `PATCH /v1/admin/tenants/{id}` — dashboard shows cascade preview before confirming |

Force-release uses dual authentication — the dashboard's nginx routes `/v1/reservations*` to `cycles-server:7878` and the runtime server validates both keys before executing. The audit log tags the action with `metadata.actor_type=admin_on_behalf_of`. See [Force-Releasing Stuck Reservations](/how-to/force-releasing-stuck-reservations-as-an-operator) for the underlying flow.

## Events investigation

The Events page is correlation-first, not time-first:

- Event rows carry a `correlation_id` when the emitting service populates one, plus `request_id` for the originating HTTP request. The current reference runtime leaves `correlation_id` absent on its implemented event paths; selected admin lifecycle, bulk, and cascade operations populate server-composed values. Clicking a present identifier filters to related events; audit rows join via `trace_id`/`request_id` rather than `correlation_id`.
- Expandable detail rows show the full event payload — including `data`, `actor`, `metadata`, and delivery outcome if the event went out over a webhook.
- Filters: event type, category, tenant, scope, time range, correlation ID.

Events poll every 15 seconds (the most aggressive of any page) because incident response typically starts here.

Per-row **Copy JSON** (v0.1.25.37+) is available on every surface rendering an event, audit entry, event-timeline entry, or webhook delivery — part of the shared triage affordances extracted to the icon library in v0.1.25.40. Correlation chips render per row type: `trace_id`/`request_id` on Audit rows; `correlation_id` additionally on Event and event-timeline rows when present. Delivery rows carry `trace_id` in the export rather than a chip.

## Audit page

Audit is the one page that is manual-only. You build a query, press **Run Query**, and the dashboard calls `GET /v1/admin/audit/logs` with the filter you built.

Supported filters (v0.1.25.33 UI + v0.1.25.27 server DSL): `tenant_id`, `key_id`, `operation` (IN-list), `resource_type` (typeahead + IN-list), `resource_id`, `request_id`, `trace_id`, `error_code` (IN-list), `error_code_exclude` (NOT-IN-list), `status_min` / `status_max` (range), free-text `search`, and time range (`from` / `to`). Deep-link URL params (`?error_code_exclude=`, `?status_min=`) support sharable filter state. Results can be exported as CSV or JSON for compliance review.

Metadata fields such as `metadata.actor_type`, bulk-action `metadata.idempotency_key`, and per-row bulk outcomes are displayed in the expanded row and exported JSON, but they are not standalone server-side query parameters. Use top-level filters like `operation`, `tenant_id`, `resource_type`, `resource_id`, `trace_id`, or `request_id` to narrow the result set before inspecting metadata.

Bulk-action audit rows expand into a structured detail panel (v0.1.25.38) that renders `succeeded_ids`, `failed_rows`, `skipped_rows`, `filter` echo, and `duration_ms` as a first-class layout instead of raw JSON — per-row copy affordances are wired for immediate triage.

Failed-request entries (added in `cycles-server-admin` v0.1.25.20) are included in results. In v0.1.25.28+ servers, their `tenant_id` is `__unauth__` (pre-auth failures) or `__admin__` (admin-plane ops); pre-.28 rows continue to show the historical `<unauthenticated>` literal. All three are queryable from the Audit filter dropdown. Tiered retention — authenticated entries live 400 days by default, unauthenticated entries 30 days.

## Monitoring the dashboard itself

The dashboard container is nginx serving a static SPA plus a reverse proxy — its own liveness check is `GET /` (returns the SPA shell; this is what the container's Docker `HEALTHCHECK` probes). Backend health is the health of `cycles-server-admin`. Two good synthetic monitoring targets:

- `GET /v1/admin/overview` with the `X-Admin-API-Key` header — requires the admin key, but if it returns 200 the full stack (Redis + admin + auth) is working.
- `GET /actuator/health/readiness` on the admin server — the unauthenticated Redis-aware readiness probe; the bundled compose files use it as the healthcheck for the admin, runtime, and events services.

Alert on the overview payload's `failing_webhooks` and `over_limit_scopes` arrays. On servers that populate v0.1.26 action-governance fields, also alert on counters at limit in `quota_health` and on spikes in `recent_denials_by_reason.ACTION_QUOTA_EXCEEDED`, `recent_denials_by_reason.ACTION_KIND_DENIED`, or `recent_denials_by_reason.ACTION_KIND_NOT_ALLOWED`.

## Mobile layout (v0.1.25.58)

The dashboard is admin-console density, not phone-native, but v0.1.25.58 landed a mobile-responsive sweep covering the paths operators actually take from a phone during incident response:

- Shell: Escape closes the drawer with focus-return to the hamburger; hamburger is sized 44×44 with `aria-expanded` / `aria-controls`; root uses `h-dvh` so mobile Safari's collapsing URL bar doesn't cut off content.
- Layout: `PageHeader` reflows to a column on narrow viewports; `LoginView` and `NotFoundView` fit 320-wide screens.
- Menus and dialogs: `RowActionsMenu` clamps horizontally to the viewport; `FormDialog` and `ConfirmAction` footers flex-wrap so buttons don't clip.
- Tables: minimum widths tightened (AuditView 1000 → 900 px), with horizontal scroll as the fallback when rows don't fit.

**Known deferrals.** Virtualized list tables still use horizontal scroll on phones rather than a card layout; the command palette's soft-keyboard viewport handling is not wired to `visualViewport`; the bulk-action preview / result dialog tables overflow on narrow viewports; the `TimeRangePicker` popover can overflow horizontally. None block incident triage — they're follow-ups for a dedicated mobile pass.

## Next steps

- [Deploy the Cycles Admin Dashboard](/quickstart/deploying-the-cycles-dashboard) — deployment, routing, and hardening
- [Using Bulk Actions](/how-to/using-bulk-actions-for-tenants-and-webhooks) — the API behind the bulk lanes
- [Force-Releasing Stuck Reservations](/how-to/force-releasing-stuck-reservations-as-an-operator) — runtime-plane incident response
- [Rolling Over Billing Periods with RESET_SPENT](/how-to/rolling-over-billing-periods-with-reset-spent) — the funding operation behind the Budgets page dropdown
- [Admin API reference](/admin-api/) — the endpoints every dashboard page calls



# Webhook Integrations

Cycles emits webhook events from implemented budget, reservation, API-key, tenant, and other lifecycle hooks. The schema also registers planned event types that are not emitted yet; check the [Event Payloads Reference](/protocol/event-payloads-reference) for current status. This guide shows concrete payload examples and integrations with common services.

::: info How webhooks are delivered
Webhook subscriptions are **configured** via the Admin Server (port 7979). Events are **delivered** by the [Cycles Events Service](/quickstart/deploying-the-events-service) — a separate, optional outbound worker that consumes from Redis and posts to your endpoints with HMAC-SHA256 signatures. The Events Service must be deployed for webhook delivery to work.
:::

## Webhook Payload Examples

### reservation.denied

Emitted when a dry-run reservation or `/v1/decide` evaluation returns `DENY` (budget exceeded, overdraft limit, etc.). The current live reservation error path returns a 4xx response and does not emit this event; monitor the runtime denial counter or application errors for live failures.

```json
{
  "event_id": "evt_a1b2c3d4e5f67890",
  "event_type": "reservation.denied",
  "category": "reservation",
  "timestamp": "2026-04-01T14:32:01.456Z",
  "tenant_id": "acme-corp",
  "scope": "tenant:acme-corp/workspace:prod/agent:support-bot",
  "actor": {
    "type": "api_key",
    "key_id": "key_9f8e7d6c-5b4a-3210"
  },
  "source": "cycles-server",
  "data": {
    "scope": "tenant:acme-corp/workspace:prod/agent:support-bot",
    "reason_code": "BUDGET_EXCEEDED",
    "requested_amount": 5000000
  },
  "request_id": "req_abc123"
}
```

### budget.exhausted

Emitted once when remaining budget transitions from above zero to zero.

```json
{
  "event_id": "evt_f0e1d2c3b4a59687",
  "event_type": "budget.exhausted",
  "category": "budget",
  "timestamp": "2026-04-01T14:32:00.123Z",
  "tenant_id": "acme-corp",
  "scope": "tenant:acme-corp/workspace:prod",
  "actor": {
    "type": "api_key",
    "key_id": "key_9f8e7d6c-5b4a-3210"
  },
  "source": "cycles-server",
  "data": {
    "scope": "tenant:acme-corp/workspace:prod",
    "unit": "USD_MICROCENTS",
    "threshold": 1.0,
    "utilization": 1.0,
    "allocated": 10000000,
    "remaining": 0,
    "spent": 9000000,
    "reserved": 1000000,
    "direction": "rising"
  }
}
```

The payload is the balance snapshot that caused the transition; the envelope identifies the tenant, scope, and actor. Query the balance API before remediation because the ledger may have changed since emission.

### reservation.commit_overage

Emitted when a commit's actual amount exceeds its reservation estimate. The current v0.1.25.46+ runtime populates all eight data fields:

```json
{
  "event_id": "evt_1122334455667788",
  "event_type": "reservation.commit_overage",
  "category": "reservation",
  "timestamp": "2026-04-01T13:15:00.789Z",
  "tenant_id": "acme-corp",
  "scope": "tenant:acme-corp/workflow:support",
  "source": "cycles-server",
  "data": {
    "reservation_id": "res_a1b2c3d4",
    "scope": "tenant:acme-corp/workflow:support",
    "unit": "USD_MICROCENTS",
    "estimated_amount": 40000000,
    "actual_amount": 48000000,
    "overage": 8000000,
    "overage_policy": "ALLOW_IF_AVAILABLE",
    "debt_incurred": 0
  }
}
```

`budget.threshold_crossed` is registered in the governance schema but is not emitted by the current reference runtime. Build pre-exhaustion alerts from balance polling or application metrics.

### budget.over_limit_entered

Emitted when debt exceeds overdraft_limit.

```json
{
  "event_id": "evt_aabbccdd11223344",
  "event_type": "budget.over_limit_entered",
  "category": "budget",
  "timestamp": "2026-04-01T14:45:12.345Z",
  "tenant_id": "acme-corp",
  "scope": "tenant:acme-corp/workspace:prod",
  "source": "cycles-server",
  "data": {
    "scope": "tenant:acme-corp/workspace:prod",
    "unit": "USD_MICROCENTS",
    "debt": 15000000,
    "overdraft_limit": 10000000,
    "is_over_limit": true,
    "debt_utilization": 1.5
  }
}
```

### tenant.suspended

Emitted when a tenant is suspended.

```json
{
  "event_id": "evt_5566778899aabbcc",
  "event_type": "tenant.suspended",
  "category": "tenant",
  "timestamp": "2026-04-01T09:00:00.000Z",
  "tenant_id": "acme-corp",
  "source": "cycles-admin",
  "actor": {
    "type": "admin"
  },
  "data": {
    "tenant_id": "acme-corp",
    "new_status": "SUSPENDED",
    "changed_fields": ["status"]
  }
}
```

### api_key.auth_failed

Emitted when authentication fails (invalid or revoked key).

```json
{
  "event_id": "evt_ddee0011ff223344",
  "event_type": "api_key.auth_failed",
  "category": "api_key",
  "timestamp": "2026-04-01T11:22:33.456Z",
  "tenant_id": "acme-corp",
  "source": "cycles-admin",
  "data": {
    "key_id": "key_expired_abc",
    "failure_reason": "KEY_EXPIRED",
    "source_ip": "203.0.113.42"
  }
}
```

## Webhook Delivery Headers

Every webhook POST includes these headers:

```http
POST /your-webhook-endpoint HTTP/1.1
Content-Type: application/json
X-Cycles-Event-Id: evt_a1b2c3d4e5f67890
X-Cycles-Event-Type: reservation.denied
X-Cycles-Trace-Id: 4bf92f3577b34da6a3ce929d0e0e4736
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
X-Cycles-Signature: sha256=a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2
User-Agent: cycles-server-events/0.1.25.23
```

`X-Cycles-Trace-Id` and W3C `traceparent` are always present — use them to join the delivery back to the originating request across the audit log, events, and delivery records. Two headers are conditional: `X-Request-Id` is sent when the originating event carries a `request_id`, and `X-Cycles-Signature` is sent when the subscription has a signing secret. Any custom headers configured on the subscription are also included, except names that collide with the reserved set above (reserved names are ignored with a warning). The `User-Agent` version tracks the deployed events-service version.

## Signature Verification

Always verify the `X-Cycles-Signature` header before processing a webhook:

### Python

```python
import hmac
import hashlib

def verify_webhook(body: bytes, secret: str, signature: str) -> bool:
    expected = "sha256=" + hmac.new(
        secret.encode("utf-8"), body, hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, signature)

# In your Flask/FastAPI handler:
@app.post("/webhook")
async def handle_webhook(request: Request):
    body = await request.body()
    sig = request.headers.get("X-Cycles-Signature", "")
    if not verify_webhook(body, SIGNING_SECRET, sig):
        return Response(status_code=401)

    event = json.loads(body)
    event_type = event["event_type"]
    # Route to handler...
```

### Node.js

```javascript
const crypto = require('crypto');

function verifyWebhook(body, secret, signature) {
  const expected = 'sha256=' + crypto
    .createHmac('sha256', secret)
    .update(body)
    .digest('hex');
  if (expected.length !== signature.length) return false;
  return crypto.timingSafeEqual(
    Buffer.from(expected),
    Buffer.from(signature)
  );
}

// In Express:
app.post('/webhook', express.raw({ type: 'application/json' }), (req, res) => {
  const sig = req.headers['x-cycles-signature'] || '';
  if (!verifyWebhook(req.body, SIGNING_SECRET, sig)) {
    return res.status(401).send('Invalid signature');
  }

  const event = JSON.parse(req.body.toString());
  console.log(`Event: ${event.event_type} for tenant ${event.tenant_id}`);
  res.status(200).json({ received: true });
});
```

### Go

```go
import (
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
)

func verifyWebhook(body []byte, secret, signature string) bool {
    mac := hmac.New(sha256.New, []byte(secret))
    mac.Write(body)
    expected := "sha256=" + hex.EncodeToString(mac.Sum(nil))
    return hmac.Equal([]byte(expected), []byte(signature))
}
```

### Java / Spring Boot

```java
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;

@RestController
public class WebhookController {

    @Value("${cycles.webhook.signing-secret}")
    private String signingSecret;

    @PostMapping("/webhook")
    public ResponseEntity<Void> handleWebhook(
            @RequestBody byte[] body,
            @RequestHeader("X-Cycles-Signature") String signature) {

        if (!verifySignature(body, signingSecret, signature)) {
            return ResponseEntity.status(401).build();
        }

        String json = new String(body, StandardCharsets.UTF_8);
        // Parse and route event...
        return ResponseEntity.ok().build();
    }

    private boolean verifySignature(byte[] body, String secret, String signature) {
        try {
            Mac mac = Mac.getInstance("HmacSHA256");
            mac.init(new SecretKeySpec(
                secret.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
            byte[] hash = mac.doFinal(body);
            StringBuilder hex = new StringBuilder("sha256=");
            for (byte b : hash) hex.append(String.format("%02x", b));
            // Constant-time comparison to prevent timing attacks
            return MessageDigest.isEqual(
                hex.toString().getBytes(StandardCharsets.UTF_8),
                signature.getBytes(StandardCharsets.UTF_8));
        } catch (Exception e) {
            return false;
        }
    }
}
```

::: tip Raw body required
Spring Boot parses JSON by default. Use `byte[]` as the parameter type (or configure `HttpMessageConverter`) to get the raw bytes for HMAC verification. If you parse JSON first, whitespace differences will produce a different hash.
:::
## Integration: PagerDuty

Route budget alerts to PagerDuty for on-call incident response.

### Setup

```bash
# Create subscription for budget and security alert events
curl -X POST http://localhost:7979/v1/admin/webhooks \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-middleware.example.com/cycles-to-pagerduty",
    "event_types": [
      "budget.exhausted",
      "budget.over_limit_entered",
      "reservation.commit_overage",
      "reservation.denied",
      "api_key.auth_failed"
    ],
    "signing_secret": "pd-webhook-secret-abc123",
    "disable_after_failures": 20
  }'
```

### Middleware (Python)

Transform Cycles events into PagerDuty Events API v2 format:

```python
import json
import requests

PAGERDUTY_ROUTING_KEY = "your-pagerduty-integration-key"

SEVERITY_MAP = {
    "budget.exhausted": "critical",
    "budget.over_limit_entered": "critical",
    "reservation.commit_overage": "warning",
    "reservation.denied": "warning",
    "api_key.auth_failed": "info",
}

@app.post("/cycles-to-pagerduty")
async def forward_to_pagerduty(request: Request):
    body = await request.body()
    # Verify signature first (see above)

    event = json.loads(body)
    severity = SEVERITY_MAP.get(event["event_type"], "info")

    pd_payload = {
        "routing_key": PAGERDUTY_ROUTING_KEY,
        "event_action": "trigger",
        "dedup_key": event["event_id"],  # Correlates retries to the same PD alert
        "payload": {
            "summary": f"[Cycles] {event['event_type']} — tenant: {event['tenant_id']}",
            "severity": severity,
            "source": event.get("scope", event["tenant_id"]),
            "component": event["source"],
            "group": event["category"],
            "custom_details": event.get("data", {})
        }
    }

    requests.post(
        "https://events.pagerduty.com/v2/enqueue",
        json=pd_payload
    )
    return {"ok": True}
```

### What triggers PagerDuty alerts

| Cycles Event | PagerDuty Severity | When |
|---|---|---|
| `budget.exhausted` | Critical | Remaining reached zero; new positive reservations deriving this scope and unit may be denied |
| `budget.over_limit_entered` | Critical | Debt exceeded overdraft limit; new reservations blocked until debt repaid |
| `reservation.commit_overage` | Warning | Actual usage exceeded the reservation estimate |
| `reservation.denied` | Warning | A dry-run or decide evaluation would deny |

## Integration: Slack

Post budget notifications to a Slack channel.

### Setup

```bash
# Subscribe to specific budget and tenant alert events
curl -X POST http://localhost:7979/v1/admin/webhooks \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-middleware.example.com/cycles-to-slack",
    "event_types": [
      "reservation.commit_overage",
      "budget.exhausted",
      "budget.over_limit_entered",
      "budget.funded",
      "reservation.denied",
      "tenant.suspended",
      "tenant.closed"
    ],
    "signing_secret": "slack-webhook-secret-xyz"
  }'
```

> **Note:** `event_categories` is additive with `event_types`. If you specify `"event_categories": ["budget"]`, you receive **all** `budget.*` events (17 types including `budget.created`, `budget.debited`, `budget.closed_via_tenant_cascade`, etc.), not just the ones in `event_types`. Use `event_types` alone when you want precise control over which events trigger notifications.

### Middleware (Node.js)

```javascript
const SLACK_WEBHOOK_URL = 'https://hooks.slack.com/services/T.../B.../xxx';

const EMOJI = {
  'budget.exhausted': ':rotating_light:',
  'budget.over_limit_entered': ':no_entry:',
  'reservation.commit_overage': ':warning:',
  'budget.funded': ':money_with_wings:',
  'reservation.denied': ':no_entry:',
  'tenant.suspended': ':pause_button:',
  'tenant.closed': ':stop_sign:',
};

// Format amount based on unit type (protocol supports multiple units)
function formatAmount(amount, unit) {
  switch (unit) {
    case 'USD_MICROCENTS': return `$${(amount / 100000000).toFixed(2)}`;
    case 'TOKENS':         return `${amount.toLocaleString()} tokens`;
    case 'CREDITS':        return `${amount.toLocaleString()} credits`;
    case 'RISK_POINTS':    return `${amount.toLocaleString()} risk points`;
    default:               return `${amount.toLocaleString()} ${unit || 'units'}`;
  }
}

app.post('/cycles-to-slack', express.raw({ type: 'application/json' }), async (req, res) => {
  // Verify signature first (see Signature Verification above)

  const event = JSON.parse(req.body.toString());
  const emoji = EMOJI[event.event_type] || ':bell:';
  const data = event.data || {};

  let text = `${emoji} *${event.event_type}*\n`;
  text += `Tenant: \`${event.tenant_id}\`\n`;
  if (event.scope) text += `Scope: \`${event.scope}\`\n`;

  if (data.utilization !== undefined) {
    text += `Utilization: ${(data.utilization * 100).toFixed(1)}%\n`;
  }
  if (data.remaining !== undefined) {
    text += `Remaining: ${formatAmount(data.remaining, data.unit)}\n`;
  }
  if (data.reason_code) {
    text += `Reason: ${data.reason_code}\n`;
  }
  if (data.estimated_amount !== undefined) {
    text += `Estimated: ${formatAmount(data.estimated_amount, data.unit)}\n`;
  }
  if (data.actual_amount !== undefined) {
    text += `Actual: ${formatAmount(data.actual_amount, data.unit)}\n`;
  }
  if (data.overage !== undefined) {
    text += `Overage: ${formatAmount(data.overage, data.unit)}\n`;
  }

  await fetch(SLACK_WEBHOOK_URL, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      text,
      unfurl_links: false,
    }),
  });

  res.status(200).json({ ok: true });
});
```

### Example Slack messages

```
:warning: reservation.commit_overage
Tenant: `acme-corp`
Scope: `tenant:acme-corp/workflow:support`
Estimated: $0.40
Actual: $0.48
Overage: $0.08

:rotating_light: budget.exhausted
Tenant: `acme-corp`
Scope: `tenant:acme-corp/workspace:prod`
Utilization: 100.0%
Remaining: $0.00

:no_entry: reservation.denied
Tenant: `acme-corp`
Scope: `tenant:acme-corp/workspace:prod/agent:support-bot`
Reason: BUDGET_EXCEEDED
```

## Integration: ServiceNow

Create incidents in ServiceNow for critical budget events.

### Setup

```bash
curl -X POST http://localhost:7979/v1/admin/webhooks \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-middleware.example.com/cycles-to-servicenow",
    "event_types": [
      "budget.over_limit_entered",
      "budget.exhausted",
      "api_key.auth_failed"
    ],
    "signing_secret": "snow-secret-123"
  }'
```

### Middleware (Python)

```python
import hmac
import hashlib
import json
import requests

SNOW_INSTANCE = "yourcompany.service-now.com"
SNOW_USER = "cycles-integration"
SNOW_PASS = "..."
SIGNING_SECRET = "snow-secret-123"

PRIORITY_MAP = {
    "budget.over_limit_entered": "2",   # High
    "budget.exhausted": "2",            # High
    "api_key.auth_failed": "2",           # High
}

@app.post("/cycles-to-servicenow")
async def forward_to_snow(request: Request):
    body = await request.body()
    sig = request.headers.get("X-Cycles-Signature", "")
    expected = "sha256=" + hmac.new(
        SIGNING_SECRET.encode("utf-8"), body, hashlib.sha256
    ).hexdigest()
    if not hmac.compare_digest(expected, sig):
        return Response(status_code=401)

    event = json.loads(body)

    # NOTE: caller_id and assignment_group are reference fields. The values
    # below use display values, which requires sysparm_input_display_value=true.
    # For production, use sys_id values instead (e.g., "caller_id": "6816f79cc0a8016401c5a33be04be441")
    # or configure the API call with the display_value parameter.
    incident = {
        "short_description": f"Cycles: {event['event_type']} — {event['tenant_id']}",
        "description": json.dumps(event, indent=2),
        "urgency": PRIORITY_MAP.get(event["event_type"], "3"),
        "category": "Software",
        "subcategory": "Budget Governance",
        "caller_id": "cycles-system",
        "assignment_group": "Platform Engineering",
        "work_notes": f"Cycles event_id: {event['event_id']}\nCategory: {event['category']}",
    }

    requests.post(
        f"https://{SNOW_INSTANCE}/api/now/table/incident",
        json=incident,
        auth=(SNOW_USER, SNOW_PASS),
        headers={"Content-Type": "application/json"},
        params={"sysparm_input_display_value": "true"}  # Allows display names for reference fields
    )
    return {"ok": True}
```

## Integration: Datadog

Post budget events as Datadog Events for correlation with infrastructure metrics.

### Setup

```bash
curl -X POST http://localhost:7979/v1/admin/webhooks \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-middleware.example.com/cycles-to-datadog",
    "event_types": [
      "budget.exhausted",
      "budget.over_limit_entered",
      "reservation.commit_overage",
      "reservation.denied"
    ],
    "signing_secret": "dd-webhook-secret"
  }'
```

### Middleware (Python)

```python
import hmac
import hashlib
import json
import requests

DD_API_KEY = "your-datadog-api-key"
SIGNING_SECRET = "dd-webhook-secret"

ALERT_TYPE_MAP = {
    "budget.exhausted": "error",
    "budget.over_limit_entered": "error",
    "reservation.commit_overage": "warning",
    "reservation.denied": "warning",
}

@app.post("/cycles-to-datadog")
async def forward_to_datadog(request: Request):
    body = await request.body()
    sig = request.headers.get("X-Cycles-Signature", "")
    expected = "sha256=" + hmac.new(
        SIGNING_SECRET.encode("utf-8"), body, hashlib.sha256
    ).hexdigest()
    if not hmac.compare_digest(expected, sig):
        return Response(status_code=401)

    event = json.loads(body)
    data = event.get("data", {})

    dd_event = {
        "title": f"Cycles: {event['event_type']}",
        "text": f"Tenant: {event['tenant_id']}\nScope: {event.get('scope', 'N/A')}\nSource: {event['source']}",
        "alert_type": ALERT_TYPE_MAP.get(event["event_type"], "info"),
        "source_type_name": "cycles",
        "tags": [
            f"tenant:{event['tenant_id']}",
            f"event_type:{event['event_type']}",
            f"category:{event['category']}",
            f"source:{event['source']}",
        ],
    }

    if data.get("utilization") is not None:
        dd_event["text"] += f"\nUtilization: {data['utilization'] * 100:.1f}%"
    if data.get("scope"):
        dd_event["tags"].append(f"scope:{data['scope']}")

    requests.post(
        "https://api.datadoghq.com/api/v1/events",
        json=dd_event,
        headers={
            "DD-API-KEY": DD_API_KEY,
            "Content-Type": "application/json",
        },
    )
    return {"ok": True}
```

### Event overlays in Datadog

Budget events posted via the Events API appear in Datadog's [Events Explorer](https://docs.datadoghq.com/events/explorer/) and can be overlaid on Datadog dashboards. Use `tags` for filtering — e.g., show only `budget.exhausted` events on your cost dashboard.

## Integration: Microsoft Teams

Post budget alerts to a Teams channel using incoming webhooks.

### Setup

```bash
curl -X POST http://localhost:7979/v1/admin/webhooks \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-middleware.example.com/cycles-to-teams",
    "event_types": [
      "budget.exhausted",
      "budget.over_limit_entered",
      "reservation.commit_overage",
      "reservation.denied",
      "tenant.suspended"
    ],
    "signing_secret": "teams-webhook-secret"
  }'
```

### Middleware (Python)

Transform Cycles events into [Adaptive Card](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/connectors-using) format for Teams:

> **Note:** Microsoft says Microsoft 365 Connectors are [nearing deprecation](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/connectors-using) and recommends the Workflows app going forward. Incoming Webhooks and Adaptive Card posting are still documented and functional, but new development should prefer [Power Automate Workflows](https://learn.microsoft.com/en-us/power-automate/teams/send-a-message-in-teams) where possible. The Adaptive Card payload format below works with both approaches.

```python
import hmac
import hashlib
import json
import requests

TEAMS_WEBHOOK_URL = "https://your-org.webhook.office.com/webhookb2/..."  # Legacy connector
# Or Power Automate Workflow HTTP trigger URL
SIGNING_SECRET = "teams-webhook-secret"

CARD_COLOR_MAP = {
    "budget.exhausted": "attention",       # Red
    "budget.over_limit_entered": "attention",
    "reservation.commit_overage": "warning", # Yellow
    "reservation.denied": "warning",
    "tenant.suspended": "accent",           # Blue
}

def format_amount(amount, unit):
    if unit == "USD_MICROCENTS":
        return f"${amount / 100_000_000:.2f}"
    return f"{amount:,} {unit.lower()}" if unit else f"{amount:,}"

@app.post("/cycles-to-teams")
async def forward_to_teams(request: Request):
    body = await request.body()
    sig = request.headers.get("X-Cycles-Signature", "")
    expected = "sha256=" + hmac.new(
        SIGNING_SECRET.encode("utf-8"), body, hashlib.sha256
    ).hexdigest()
    if not hmac.compare_digest(expected, sig):
        return Response(status_code=401)

    event = json.loads(body)
    data = event.get("data", {})
    color = CARD_COLOR_MAP.get(event["event_type"], "default")

    facts = [
        {"title": "Tenant", "value": event["tenant_id"]},
        {"title": "Event", "value": event["event_type"]},
        {"title": "Source", "value": event["source"]},
    ]
    if event.get("scope"):
        facts.append({"title": "Scope", "value": event["scope"]})
    if data.get("utilization") is not None:
        facts.append({"title": "Utilization", "value": f"{data['utilization'] * 100:.1f}%"})
    if data.get("remaining") is not None:
        facts.append({"title": "Remaining", "value": format_amount(data["remaining"], data.get("unit"))})
    if data.get("reason_code"):
        facts.append({"title": "Reason", "value": data["reason_code"]})
    if data.get("estimated_amount") is not None:
        facts.append({"title": "Estimated", "value": format_amount(data["estimated_amount"], data.get("unit"))})
    if data.get("actual_amount") is not None:
        facts.append({"title": "Actual", "value": format_amount(data["actual_amount"], data.get("unit"))})
    if data.get("overage") is not None:
        facts.append({"title": "Overage", "value": format_amount(data["overage"], data.get("unit"))})

    card = {
        "type": "message",
        "attachments": [{
            "contentType": "application/vnd.microsoft.card.adaptive",
            "content": {
                "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
                "type": "AdaptiveCard",
                "version": "1.5",
                "body": [
                    {
                        "type": "TextBlock",
                        "size": "medium",
                        "weight": "bolder",
                        "text": f"Cycles: {event['event_type']}",
                        "color": color,
                    },
                    {
                        "type": "FactSet",
                        "facts": facts,
                    },
                ],
            },
        }],
    }

    requests.post(TEAMS_WEBHOOK_URL, json=card)
    return {"ok": True}
```

### Example Teams card

The card renders as a structured fact table showing the event type, tenant, source service, scope path, and the event's populated data fields.

```
┌─────────────────────────────────────┐
│ ⚠ Cycles: reservation.commit_overage│
│                                     │
│ Tenant:      acme-corp              │
│ Event:       reservation.commit_overage│
│ Source:      cycles-server           │
│ Scope:       tenant:acme-corp/...   │
│ Estimated:   $0.40                  │
│ Actual:      $0.48                  │
│ Overage:     $0.08                  │
└─────────────────────────────────────┘
```

## Integration: Opsgenie

Route alerts to Opsgenie for on-call management (popular with Atlassian/Jira teams).

### Setup

```bash
curl -X POST http://localhost:7979/v1/admin/webhooks \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-middleware.example.com/cycles-to-opsgenie",
    "event_types": [
      "budget.exhausted",
      "budget.over_limit_entered",
      "reservation.denied",
      "api_key.auth_failed"
    ],
    "signing_secret": "og-webhook-secret"
  }'
```

### Middleware (Python)

```python
import hmac
import hashlib
import json
import requests

OPSGENIE_API_KEY = "your-opsgenie-api-key"
SIGNING_SECRET = "og-webhook-secret"

PRIORITY_MAP = {
    "budget.exhausted": "P2",
    "budget.over_limit_entered": "P1",
    "api_key.auth_failed": "P2",
    "reservation.denied": "P3",
}

@app.post("/cycles-to-opsgenie")
async def forward_to_opsgenie(request: Request):
    body = await request.body()
    sig = request.headers.get("X-Cycles-Signature", "")
    expected = "sha256=" + hmac.new(
        SIGNING_SECRET.encode("utf-8"), body, hashlib.sha256
    ).hexdigest()
    if not hmac.compare_digest(expected, sig):
        return Response(status_code=401)

    event = json.loads(body)

    alert = {
        "message": f"Cycles: {event['event_type']} — {event['tenant_id']}",
        "alias": event["event_id"],  # Dedup key — same event won't create duplicate alerts
        "description": json.dumps(event, indent=2),
        "priority": PRIORITY_MAP.get(event["event_type"], "P3"),
        "source": event["source"],
        "tags": [event["category"], event["tenant_id"]],
        "entity": event.get("scope", event["tenant_id"]),
        "details": event.get("data", {}),
    }

    requests.post(
        "https://api.opsgenie.com/v2/alerts",
        json=alert,
        headers={
            "Authorization": f"GenieKey {OPSGENIE_API_KEY}",
            "Content-Type": "application/json",
        },
    )
    return {"ok": True}
```

> **Note:** Opsgenie uses `alias` for deduplication — setting it to `event_id` ensures retried webhook deliveries don't create duplicate alerts.

## Integration: Custom Receiver (Direct)

For simple use cases, receive webhooks directly without middleware.

**Best practices for webhook receivers:**
- **Acknowledge quickly** — return `200 OK` as fast as possible. The events service treats non-2xx as failure and will retry with exponential backoff.
- **Queue internally** — if processing takes time, accept the event, enqueue it in your own job queue, and return 200 immediately.
- **Make handlers idempotent** — delivery is at-least-once, so you may receive the same event more than once. Use `event_id` (via `X-Cycles-Event-Id` header) for deduplication.
- **Verify signatures** — always check `X-Cycles-Signature` before processing. Never trust unverified payloads.

```python
from flask import Flask, request
import hmac, hashlib, json

app = Flask(__name__)
SIGNING_SECRET = "your-signing-secret"

@app.post("/webhook")
def handle():
    # 1. Verify signature
    body = request.get_data()
    sig = request.headers.get("X-Cycles-Signature", "")
    expected = "sha256=" + hmac.new(
        SIGNING_SECRET.encode("utf-8"), body, hashlib.sha256
    ).hexdigest()
    if not hmac.compare_digest(expected, sig):
        return "Unauthorized", 401

    # 2. Parse event
    event = json.loads(body)
    event_type = event["event_type"]
    event_id = request.headers.get("X-Cycles-Event-Id")

    # 3. Deduplicate (at-least-once delivery)
    if already_processed(event_id):
        return "OK", 200

    # 4. Route by event type
    if event_type == "budget.exhausted":
        handle_budget_exhausted(event)
    elif event_type == "reservation.denied":
        handle_denial(event)
    elif event_type == "reservation.commit_overage":
        handle_commit_overage(event)

    mark_processed(event_id)
    return "OK", 200
```

## Tenant Self-Service Webhooks

Tenants can manage their own webhooks (restricted to `budget.*`, `reservation.*`, `tenant.*` events — 29 of 51 registered types, including the `_via_tenant_cascade` fan-out events the admin server emits in those categories on tenant close — see [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics)). Admin-only events (`api_key.*`, `policy.*`, `webhook.*`, `system.*`) are not available to tenants — a tenant-owned subscription can neither carry them nor receive them from the event stream (governance WEBHOOK SUBSCRIPTION INVARIANT 2, enforced at write, dispatch, and last-mile delivery as of cycles-server-admin 0.1.25.51 + cycles-server-events 0.1.25.23; issue #209). The one exception is the owner-triggered `/test` probe (a synthetic `system.webhook_test` ping). This holds regardless of who created the subscription — a tenant-owned row created via the admin plane or admin-on-behalf-of is bound by the same rule. To monitor a specific tenant's admin-only events, use a `__system__`-owned subscription with client-side `tenant_id` filtering — see [Tenant-accessible events](/protocol/webhook-event-delivery-protocol#tenant-accessible-events).

**Required API key permissions:**
- `webhooks:write` — create, update, delete, and test subscriptions
- `webhooks:read` — list subscriptions and delivery history
- `events:read` — query tenant's event stream via `GET /v1/events`

If the API key lacks these permissions, the server returns `403 INSUFFICIENT_PERMISSIONS`.

```bash
# Tenant creates their own webhook using their API key
curl -X POST http://localhost:7979/v1/webhooks \
  -H "X-Cycles-API-Key: $TENANT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://acme-corp.example.com/budget-alerts",
    "event_types": [
      "reservation.commit_overage",
      "budget.exhausted",
      "reservation.denied"
    ]
  }'

# Response includes the signing_secret (returned ONCE — store it securely):
# {
#   "subscription": { "subscription_id": "whsub_abc123...", ... },
#   "signing_secret": "whsec_dGVzdC1zZWNyZXQ..."
# }
```

## Webhook URL Security

The events service applies a delivery-time SSRF baseline even when the admin-configured CIDR list is empty:

- **Blocked by default:** `0.0.0.0/8`, `10.0.0.0/8`, `100.64.0.0/10`, `127.0.0.0/8`, `169.254.0.0/16`, `172.16.0.0/12`, `192.168.0.0/16`, `::1/128`, `fe80::/10`, and `fc00::/7`. Any-local and unspecified addresses are also rejected.
- **HTTPS required** unless the admin webhook-security configuration explicitly sets `allow_http: true`.
- **Admin CIDR blocks are additive.** Clearing `blocked_cidr_ranges` does not remove the events-service baseline.

Local development requires both controls below. Restart the events service after setting its environment variable:

```bash
# Events service only — development/testing escape hatch
export WEBHOOK_URL_GUARD_ALLOW_PRIVATE_NETWORKS=true

# Admin service — allow an HTTP target
curl -X PUT http://localhost:7979/v1/admin/config/webhook-security \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"allow_http": true, "blocked_cidr_ranges": []}'
```

Never set `WEBHOOK_URL_GUARD_ALLOW_PRIVATE_NETWORKS=true` in production. `allowed_url_patterns` can narrow delivery to approved public destinations, but it does not bypass private-address blocking:

```bash
curl -X PUT http://localhost:7979/v1/admin/config/webhook-security \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "allowed_url_patterns": ["https://hooks.example.com/cycles/*"],
    "blocked_cidr_ranges": ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]
  }'
```

## Event Type Reference

| Event Type | Produced By | `source` Field | Use Case |
|---|---|---|---|
| `budget.exhausted` | Runtime server | `cycles-server` | Critical: remaining reached zero on the affected ledger |
| `budget.over_limit_entered` | Runtime server | `cycles-server` | Critical: debt exceeded overdraft limit; new reservations blocked |
| `budget.debt_incurred` | Runtime server | `cycles-server` | Info: a commit or direct debit created debt via ALLOW_WITH_OVERDRAFT |
| `reservation.denied` | Runtime server | `cycles-server` | Calibration: a dry-run or decide evaluation returned DENY |
| `reservation.commit_overage` | Runtime server | `cycles-server` | Info: actual spend exceeded estimated amount |
| `reservation.expired` | Runtime server (expiry sweep) | `cycles-server` | Info: reservation TTL expired without commit/release |
| `tenant.suspended` | Admin server | `cycles-admin` | Alert: tenant operations paused |
| `tenant.closed` | Admin server | `cycles-admin` | Alert: tenant permanently closed |
| `api_key.auth_failed` | Admin server | `cycles-admin` | Security: authentication failure |
| `api_key.revoked` | Admin server | `cycles-admin` | Security: key access removed |
| `webhook.disabled` | Admin/events services | `cycles-admin` or `cycles-events` | Alert: a webhook was disabled manually or after delivery failures |
| `system.webhook_delivery_failed` | Events service | `cycles-events` | Meta: webhook delivery permanently failed after all retries |

## Next steps

- [Managing Webhooks](/how-to/managing-webhooks) — create, test, replay, and monitor webhook subscriptions
- [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol) — full 51-event-type catalog, delivery headers, retry policy, and status lifecycle
- [Deploying the Events Service](/quickstart/deploying-the-events-service) — deploy the async webhook delivery service
- [Security](/security#webhook-security) — SSRF protection, signing secret encryption, and deduplication

---

# Incident Patterns



# Concurrent Agent Overspend

A failure mode where multiple agents sharing a budget each pass local checks but collectively exceed the limit.

## The incident

A platform runs 5 agents concurrently, all spending against the same team budget of $10. Each agent checks the remaining balance before making a call and sees $8 remaining. All 5 proceed simultaneously, each spending $3. Total spend: $15 — exceeding the $10 budget by 50%.

### The race condition

```
Time 0: Budget = $10.00
Agent A checks balance → $10.00 remaining → proceeds
Agent B checks balance → $10.00 remaining → proceeds
Agent C checks balance → $10.00 remaining → proceeds
Agent D checks balance → $10.00 remaining → proceeds
Agent E checks balance → $10.00 remaining → proceeds

All 5 agents call LLM simultaneously, each spending ~$3.00

Time 1: Budget = $10.00 - $15.00 = -$5.00 (overspent)
```

### Why read-then-act doesn't work

The check-then-spend pattern is a classic [TOCTOU](https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use) race. The balance read is stale by the time the spend occurs. This is true even with database transactions — unless the balance check and the deduction are atomic.

### Without Cycles

Application-level balance checks are not concurrency-safe. Even "careful" implementations using database locks often miss edge cases under high concurrency.

### With Cycles

Cycles reservation is **atomically concurrency-safe**. Each reservation locks the requested amount across all affected scopes in a single Redis Lua script. No partial locks, no race conditions:

```
Time 0: Budget = $10.00
Agent A reserves $3.00 → ALLOW ($7.00 remaining, $3.00 reserved)
Agent B reserves $3.00 → ALLOW ($4.00 remaining, $6.00 reserved)
Agent C reserves $3.00 → ALLOW ($1.00 remaining, $9.00 reserved)
Agent D reserves $3.00 → REJECTED — HTTP 409 BUDGET_EXCEEDED (only $1.00 remaining)
Agent E reserves $3.00 → REJECTED — HTTP 409 BUDGET_EXCEEDED (only $1.00 remaining)
```

Agents D and E are rejected *before any LLM call is made*. The budget is not exceeded. On the wire, a live reserve failure is an HTTP `409` response with `error: BUDGET_EXCEEDED`; the `DENY` decision value appears only on dry-run and `decide` responses, which return `200` with the decision in the body.

### Python example

```python
import os

from runcycles import (
    BudgetExceededError,
    CyclesClient,
    CyclesConfig,
    cycles,
    set_default_client,
)

set_default_client(CyclesClient(CyclesConfig(
    base_url=os.environ["CYCLES_BASE_URL"],
    api_key=os.environ["CYCLES_API_KEY"],
    tenant="acme-corp",
    workspace="prod",
)))

@cycles(
    estimate=3_000_000,
    action_kind="llm.completion",
    action_name="gpt-4o",
    # Subject callables receive the decorated function's *args/**kwargs
    agent=lambda *args, **kwargs: kwargs.get("agent_id"),
)
def call_llm_safe(prompt: str, agent_id: str) -> str:
    return call_llm(prompt)

def agent_task(agent_id: str, task: str):
    try:
        return call_llm_safe(task, agent_id=agent_id)
    except BudgetExceededError:
        return fallback_response(task)
```

### TypeScript example

```typescript
import {
  BudgetExceededError,
  CyclesClient,
  CyclesConfig,
  setDefaultClient,
  withCycles,
} from "runcycles";

setDefaultClient(new CyclesClient(new CyclesConfig({
  baseUrl: process.env.CYCLES_BASE_URL!,
  apiKey: process.env.CYCLES_API_KEY!,
  tenant: "acme-corp",
  workspace: "prod",
})));

const callLlmSafe = withCycles(
  {
    estimate: 3_000_000,
    actionKind: "llm.completion",
    actionName: "gpt-4o",
    tenant: "acme-corp",
    workspace: "prod",
  },
  async (prompt: string): Promise<string> => {
    return await callLlm(prompt);
  }
);

async function agentTask(agentId: string, task: string): Promise<string> {
  try {
    return await callLlmSafe(task);
  } catch (err) {
    if (err instanceof BudgetExceededError) {
      return fallbackResponse(task);
    }
    throw err;
  }
}

// Run 5 agents concurrently — reservations serialize estimate admission
const results = await Promise.all(
  agents.map((agent) => agentTask(agent.id, agent.task))
);
```

## Severity and impact

Concurrent overspend is proportional to the number of parallel agents and the cost per operation. The worst case is `N agents * cost per call` overshoot.

**Concrete examples:**

| Agents | Budget | Cost per call | Overspend (no Cycles) | With Cycles |
|--------|--------|---------------|----------------------|-------------|
| 5      | $10    | $3.00         | $5.00 (50%)          | $0.00       |
| 10     | $50    | $8.00         | $30.00 (60%)         | $0.00       |
| 50     | $100   | $5.00         | $150.00 (150%)       | $0.00       |
| 100    | $500   | $10.00        | $500.00 (100%)       | $0.00       |

The overspend percentage increases with concurrency. At 100 agents each spending $10, the theoretical maximum overshoot is $500 — a full doubling of the budget.

**Compounding effect with retries.** If each agent also retries failed calls (see [Retry Storms](/incidents/retry-storms-and-idempotency-failures)), the multiplication compounds. 10 agents with 5 retries each can produce 50 concurrent calls against the same budget.

**Invoice shock.** Unlike a gradual budget drain, concurrent overspend happens in a burst. The budget goes from healthy to overdrawn in seconds, giving operators no time to intervene manually.

## Detection

### Querying for concurrent reservation patterns

Check how many reservations are active simultaneously for the same scope:

```bash
# Count active reservations per scope
curl -s "http://localhost:7878/v1/reservations?tenant=acme-corp&status=ACTIVE" \
  -H "X-Cycles-API-Key: $API_KEY" \
  | jq '.reservations | group_by(.scope_path) | map({scope_path: .[0].scope_path, count: length})'
```

If a single scope has many active reservations simultaneously, you have high concurrency against that budget.

### Spotting TOCTOU patterns in application code

Search your codebase for the check-then-spend anti-pattern:

```python
# ANTI-PATTERN: checking balance then spending is NOT safe
balance = get_balance(scope="tenant:acme-corp")
if balance.remaining > estimated_cost:
    # Another agent can spend between this check and the call
    result = call_llm(prompt)  # UNSAFE
```

The fix is to always use `reserve` instead of `balance` for authorization decisions.

### Checking for budget overruns

Compare spent against allocated to find scopes that exceeded their budget:

```bash
# Find scopes where spent exceeds allocated (overrun already happened)
curl -s "http://localhost:7878/v1/balances?tenant=acme-corp" \
  -H "X-Cycles-API-Key: $API_KEY" \
  | jq '.balances[] | select(.spent.amount > .allocated.amount) | {scope, allocated, spent, overshoot: (.spent.amount - .allocated.amount)}'
```

## Monitoring

### Alerting rules

::: warning Planned metrics — not yet registered
`cycles_scope_spent_total`, `cycles_scope_allocated_total`, `cycles_scope_remaining_total`, and `cycles_active_reservations_count` are on the roadmap but are not emitted by the current server builds (runtime 0.1.25.58 registers request-driven `cycles.*` counters — reservations, events, overdraft — but no balance or active-reservation gauges). Build these alerts from a balance-polling sidecar that pushes `GET /v1/balances` fields as gauges — see [Query balances for monitoring](/how-to/monitoring-and-alerting#query-balances-for-monitoring). Once the sidecar pushes `cycles_budget_spent` / `cycles_budget_allocated` / `cycles_budget_remaining` / `cycles_active_reservations`, the rules below apply as-is against your own gauge names.
:::

```yaml
# Alert when spent exceeds allocated for any scope
# Requires gauges pushed by a balance-polling sidecar (see note above)
- alert: CyclesBudgetOvershoot
  expr: |
    cycles_budget_spent > cycles_budget_allocated
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: "Budget overshoot on {{ $labels.scope }}: spent {{ $value }}"

# Alert on high concurrent reservation count (pre-incident warning)
- alert: CyclesHighConcurrentReservations
  expr: |
    cycles_active_reservations > 20
  for: 1m
  labels:
    severity: warning
  annotations:
    summary: "{{ $value }} concurrent reservations on {{ $labels.scope }}"

# Alert when remaining budget drops below 10% with active reservations
- alert: CyclesBudgetNearExhaustion
  expr: |
    cycles_budget_remaining / cycles_budget_allocated < 0.1
    and cycles_active_reservations > 0
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: "Budget nearly exhausted on {{ $labels.scope }} with active reservations"
```

For detailed monitoring setup, see [Monitoring and Alerting](/how-to/monitoring-and-alerting).

## Testing for concurrency issues

Concurrency bugs are hard to reproduce in unit tests. Use these strategies to verify your budget enforcement holds under concurrent load.

### Load test with parallel reservations

```python
import asyncio
import uuid

from runcycles import (
    Action,
    Amount,
    AsyncCyclesClient,
    CommitRequest,
    CyclesConfig,
    ReservationCreateRequest,
    Subject,
    Unit,
)

async def test_concurrent_budget_safety():
    """Verify that concurrent reservations never exceed the budget."""
    budget_allocated = 10_000_000  # 10M microcents
    cost_per_call = 3_000_000      # 3M microcents each
    num_agents = 5

    config = CyclesConfig(base_url="http://localhost:7878", api_key="test-key")
    async with AsyncCyclesClient(config) as client:

        async def agent_reserve() -> str:
            resp = await client.create_reservation(ReservationCreateRequest(
                idempotency_key=f"test-{uuid.uuid4()}",
                subject=Subject(tenant="test-tenant"),
                action=Action(kind="llm.completion", name="gpt-4o"),
                estimate=Amount(amount=cost_per_call, unit=Unit.USD_MICROCENTS),
            ))
            if resp.status == 409:  # BUDGET_EXCEEDED — denied before any spend
                return "denied"
            reservation_id = resp.body["reservation_id"]
            # Simulate work
            await asyncio.sleep(0.1)
            await client.commit_reservation(reservation_id, CommitRequest(
                idempotency_key=f"commit-{reservation_id}",
                actual=Amount(amount=cost_per_call, unit=Unit.USD_MICROCENTS),
            ))
            return "committed"

        results = await asyncio.gather(
            *[agent_reserve() for _ in range(num_agents)]
        )

    committed = results.count("committed")
    denied = results.count("denied")

    # At most 3 agents can commit (3 * 3M = 9M < 10M budget)
    assert committed <= 3, f"Too many commits: {committed}"
    assert denied >= 2, f"Expected denials, got {denied}"
    total_spent = committed * cost_per_call
    assert total_spent <= budget_allocated, f"Overspent: {total_spent}"
```

### TypeScript concurrency test

```typescript
import { randomUUID } from "node:crypto";
import { CyclesClient, CyclesConfig } from "runcycles";

async function testConcurrentBudgetSafety() {
  const budgetAllocated = 10_000_000;
  const costPerCall = 3_000_000;
  const numAgents = 5;

  const client = new CyclesClient(new CyclesConfig({
    baseUrl: "http://localhost:7878",
    apiKey: "test-key",
  }));

  const agentReserve = async (): Promise<"committed" | "denied"> => {
    // createReservation takes the raw snake_case wire body
    const resp = await client.createReservation({
      idempotency_key: `test-${randomUUID()}`,
      subject: { tenant: "test-tenant" },
      action: { kind: "llm.completion", name: "gpt-4o" },
      estimate: { amount: costPerCall, unit: "USD_MICROCENTS" },
    });
    if (resp.status === 409) return "denied"; // BUDGET_EXCEEDED
    const reservationId = resp.body!.reservation_id as string;
    await new Promise((r) => setTimeout(r, 100));
    await client.commitReservation(reservationId, {
      idempotency_key: `commit-${reservationId}`,
      actual: { amount: costPerCall, unit: "USD_MICROCENTS" },
    });
    return "committed";
  };

  const results = await Promise.all(
    Array.from({ length: numAgents }, () => agentReserve())
  );

  const committed = results.filter((r) => r === "committed").length;
  console.assert(committed <= 3, `Too many commits: ${committed}`);
  const totalSpent = committed * costPerCall;
  console.assert(totalSpent <= budgetAllocated, `Overspent: ${totalSpent}`);
}
```

For more testing patterns, see [Testing with Cycles](/how-to/testing-with-cycles).

## Key points

- **Balance reads are informational, not authoritative.** Querying `/v1/balances` tells you the current state, but it does not reserve anything. Two agents can read the same balance and both decide to spend.
- **Reservations are authoritative for estimate admission.** A successful reservation holds the submitted estimate. Other agents see the reduced remaining balance; later settlement still follows the commit overage policy.
- **The `remaining` field accounts for reservations.** It equals `allocated - spent - reserved - debt`. Active reservations reduce `remaining` even before they commit.

## Real-world scenarios

This pattern appears in:

- **Multi-agent workflows** where agents share a team or project budget
- **Webhook-triggered processing** where multiple events arrive simultaneously
- **Batch processing** with parallel workers
- **Auto-scaling** where new instances start making calls before the budget is recalculated

## Prevention

1. **Always reserve before protected spending.** Never rely on balance reads for admission. The `reserve` call is the concurrency-safe way to hold the submitted estimate; a balance read is only a snapshot.

2. **Use hierarchical scopes.** Even if agents have individual budgets, a shared parent scope acts as a hard cap. If 5 agents each have a $5 budget but the team scope is $10, the team scope prevents collective overspend:

   ```bash
   # Team-level cap
   curl -s -X POST "http://localhost:7979/v1/admin/budgets" \
     -H "Content-Type: application/json" \
     -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
     -d '{"scope": "tenant:acme-corp/workspace:prod", "unit": "USD_MICROCENTS", "allocated": {"amount": 10000000, "unit": "USD_MICROCENTS"}}'

   # Per-agent budgets (sum exceeds team cap — that's fine)
   for agent in agent-a agent-b agent-c agent-d agent-e; do
     curl -s -X POST "http://localhost:7979/v1/admin/budgets" \
       -H "Content-Type: application/json" \
       -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
       -d "{\"scope\": \"tenant:acme-corp/workspace:prod/agent:${agent}\", \"unit\": \"USD_MICROCENTS\", \"allocated\": {\"amount\": 5000000, \"unit\": \"USD_MICROCENTS\"}}"
   done
   ```

3. **Design for denial.** Agents that can't reserve budget should degrade gracefully, not crash. Return cached results, use a cheaper model, or queue the work for later. See [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) for patterns.

4. **Avoid fire-and-forget patterns.** If you spawn agents without awaiting their reservations, you lose the ability to react to denials. Always handle the reservation result before proceeding.

## Next steps

- [Idempotency, Retries and Concurrency](/concepts/idempotency-retries-and-concurrency-why-cycles-is-built-for-real-failure-modes) — how Cycles handles concurrency
- [Scope Derivation](/protocol/how-scope-derivation-works-in-cycles) — hierarchical budget enforcement
- [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) — handling denial gracefully
- [Multi-Tenant AI Cost Control](/blog/multi-tenant-ai-cost-control-per-tenant-budgets-quotas-isolation) — how per-tenant budget isolation prevents concurrent overspend across tenants



# Retry Storms and Idempotency Failures

A common failure mode in autonomous systems where retry logic multiplies cost without bound.

## The incident

An application retries failed LLM calls with exponential backoff. Each retry creates a new reservation and makes a new LLM call. When the underlying issue is transient (network blip, rate limit), retries work as intended. But when the issue is persistent (bad prompt causing errors, model returning incomplete responses that trigger re-processing), the retry loop creates unbounded spend.

### Example

An agent processes documents. When the model returns a response that fails validation, the agent retries with a modified prompt:

```python
def process_document(doc):
    for attempt in range(10):
        response = call_llm(f"Process this document: {doc}")
        if validate(response):
            return response
        # Retry with more context
        doc = doc + f"\n\nPrevious attempt failed validation. Try again."
```

Each retry calls the LLM again. With 10 retries at $0.05 per call, a single document costs $0.50 instead of $0.05. Across 1,000 documents, this turns a $50 batch into $500.

### Without Cycles

The retry loop runs all 10 attempts for every document. There's no mechanism to stop retrying when the total cost exceeds a threshold. Rate limiters don't help — each retry is a valid individual request.

### With Cycles

Each LLM call reserves budget before executing:

```python
@cycles(estimate=5000000, action_kind="llm.completion", action_name="gpt-4o")
def call_llm_guarded(prompt: str) -> str:
    return call_llm(prompt)

def process_document(doc):
    for attempt in range(10):
        try:
            response = call_llm_guarded(f"Process this document: {doc}")
            if validate(response):
                return response
        except BudgetExceededError:
            return "Document processing stopped: budget limit reached."
```

When total spend across all retries hits the budget limit, further attempts are denied immediately — no LLM call is made.

### TypeScript equivalent

Using the `runcycles` SDK, the same pattern works with `withCycles`:

```typescript
import { withCycles, BudgetExceededError } from "runcycles";

const callLlmGuarded = withCycles(
  {
    estimate: 5_000_000,
    actionKind: "llm.completion",
    actionName: "gpt-4o",
  },
  async (prompt: string): Promise<string> => {
    return await callLlm(prompt);
  }
);

async function processDocument(doc: string): Promise<string> {
  let currentDoc = doc;
  for (let attempt = 0; attempt < 10; attempt++) {
    try {
      const response = await callLlmGuarded(
        `Process this document: ${currentDoc}`
      );
      if (validate(response)) {
        return response;
      }
      currentDoc = currentDoc + "\n\nPrevious attempt failed validation. Try again.";
    } catch (err) {
      if (err instanceof BudgetExceededError) {
        return "Document processing stopped: budget limit reached.";
      }
      throw err;
    }
  }
  return "Document processing stopped: max retries reached.";
}
```

## Severity and impact

Retry storms are deceptive because each individual retry is cheap. The damage comes from multiplication across a fleet.

**Single-document cost explosion:**

| Retries per doc | Cost per call | Docs in batch | Total cost |
|-----------------|---------------|---------------|------------|
| 1 (no retries)  | $0.05         | 1,000         | $50        |
| 5               | $0.05         | 1,000         | $250       |
| 10              | $0.05         | 1,000         | $500       |
| 10              | $0.05         | 10,000        | $5,000     |

**Prompt growth makes it worse.** Each retry in the example above appends context to the prompt. By attempt #10, the prompt is significantly longer than the original. With token-based pricing, later retries cost more than earlier ones:

| Attempt | Prompt tokens | Cost per call |
|---------|--------------|---------------|
| 1       | 500          | $0.05         |
| 5       | 2,500        | $0.12         |
| 10      | 5,000        | $0.22         |

A 10-retry loop with growing prompts costs roughly $1.00 per document, not $0.50. Across 10,000 documents, that is $10,000 instead of $500.

**Fleet multiplication.** If you run 20 parallel workers processing the same batch, a retry storm in the shared batch job can multiply these figures by the worker count before any human notices the spend rate.

## Detection

### Querying for retry storm indicators

Check the ratio of active reservations to recent commits. A healthy system commits most reservations quickly. A retry storm shows many reservations being created and released (or expiring) without successful commits.

```bash
# Count active reservations for a scope
curl -s "http://localhost:7878/v1/reservations?tenant=acme-corp&status=ACTIVE" \
  -H "X-Cycles-API-Key: $API_KEY" | jq '.reservations | length'

# Check balance to see reserved vs spent
curl -s "http://localhost:7878/v1/balances?tenant=acme-corp" \
  -H "X-Cycles-API-Key: $API_KEY" | jq '.balances[] | {scope, allocated: .allocated.amount, spent: .spent.amount, reserved: .reserved.amount, remaining: .remaining.amount}'
```

If `reserved` is growing much faster than `spent`, many reservations are being created without committing — a hallmark of retry loops.

### Checking for repeated idempotency key prefixes

If you use the pattern `doc-{id}-attempt-{n}`, you can look for documents with high attempt numbers:

```bash
# List reservations and look for high attempt numbers
curl -s "http://localhost:7878/v1/reservations?tenant=acme-corp&workflow=doc-processing" \
  -H "X-Cycles-API-Key: $API_KEY" \
  | jq '[.reservations[].idempotency_key | select(test("attempt-[5-9]|attempt-[0-9]{2,}"))]'
```

Any result means at least one document hit 5+ retries.

## Monitoring

### Alerting rules

Use these Prometheus-style rules to detect retry storms before they drain budgets.

::: tip Built on the server's `cycles_*` counters
Current server builds (runtime 0.1.25.58) register operation counters under the `cycles.*` namespace — `cycles_reservations_reserve_total`, `cycles_reservations_commit_total`, `cycles_reservations_release_total`, `cycles_reservations_extend_total`, `cycles_reservations_expired_total`, `cycles_events_total`, and `cycles_overdraft_incurred_total` — tagged with `decision`, `reason`, `overage_policy`, and (toggleable) `tenant`. Spring Boot's default `http_server_requests_seconds*` histogram is also available; denied reservations return HTTP `409`, so denial spikes show up there under `status="409"`. Ratio-style rules tagged "requires sidecar" still depend on gauges pushed by a balance-polling sidecar (see [Monitoring and Alerting](/how-to/monitoring-and-alerting#balance-polling-alerts-for-signals-without-a-counter)).
:::

```yaml
# Alert when reservation rate spikes relative to commit rate.
# A ratio above 3 means most reservations are not committing — likely retries.
- alert: CyclesRetryStormDetected
  expr: |
    sum(rate(cycles_reservations_reserve_total[5m]))
      / sum(rate(cycles_reservations_commit_total[5m]))
      > 3
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "Possible retry storm: reservation/commit ratio is {{ $value }}"

# Alert when reserved amount exceeds a threshold relative to allocated.
# Requires a balance-polling sidecar to push cycles_budget_reserved / cycles_budget_allocated gauges.
- alert: CyclesHighReservedRatio
  expr: |
    cycles_budget_reserved / cycles_budget_allocated > 0.5
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Over 50% of budget is in active reservations — retries may be stacking"

# Alert when reservation denials spike (retries hitting the wall).
# A live denial is an HTTP 409 with the reason in the error code; the server also
# increments cycles_reservations_reserve_total{decision="DENY", reason=...} for it.
# Equivalent HTTP-layer form: http_server_requests_seconds_count{uri="/v1/reservations", status="409"}.
- alert: CyclesBudgetDenialSpike
  expr: |
    sum(rate(cycles_reservations_reserve_total{decision="DENY",reason=~"BUDGET_EXCEEDED|OVERDRAFT_LIMIT_EXCEEDED|BUDGET_FROZEN|BUDGET_CLOSED|DEBT_OUTSTANDING"}[5m])) > 10
  for: 1m
  labels:
    severity: warning
  annotations:
    summary: "Spike in reservation denials — retry storm may have hit budget limit"
```

### Key metrics to track

- **Reservation-to-commit ratio** per scope and per workflow. Healthy value is 1.0–1.2. Values above 2.0 indicate retries or abandoned work.
- **Mean and p99 reservation lifetime.** Retry storms produce short-lived reservations that are released (not committed) quickly.
- **Denied reservation rate.** A sudden spike in denials often means a retry storm just hit the budget ceiling.

For detailed monitoring setup, see [Monitoring and Alerting](/how-to/monitoring-and-alerting).

## Key points

- **Retries are individually valid requests.** Rate limiters can't distinguish retry #1 from retry #10.
- **Idempotency prevents double-counting.** If you use the same idempotency key for retries of the same operation, Cycles returns the original response without re-reserving. Use unique keys only for genuinely different operations.
- **Budget is the aggregate control.** Individual retries may be cheap, but their sum can be expensive. Cycles tracks the cumulative total.

## Idempotency gotcha

If your retry uses the **same idempotency key** with a **different payload** (because the prompt changed), you'll get `IDEMPOTENCY_MISMATCH`. This is correct — Cycles is telling you that this is a new operation, not a retry of the same one. Use a new idempotency key:

```python
idempotency_key = f"doc-{doc_id}-attempt-{attempt}"
```

## Prevention strategies

### 1. Per-document or per-task budget

Create a workflow-scoped budget for each document or task. All retries for that document share the same budget pool, so a single stuck document can't drain the entire batch budget:

```bash
# Create a per-document budget under the workflow scope
curl -s -X POST "http://localhost:7979/v1/admin/budgets" \
  -H "Content-Type: application/json" \
  -H "X-Cycles-API-Key: $CYCLES_API_KEY" \
  -d '{
    "scope": "tenant:acme-corp/workspace:prod/workflow:doc-processing",
    "unit": "USD_MICROCENTS",
    "allocated": {"amount": 10000000, "unit": "USD_MICROCENTS"}
  }'
```

This limits total retry spend per document to the workflow budget, regardless of how many attempts the agent makes.

### 2. Cap retries with budget checks

Before each retry, use `decide` to check if budget is available without creating a reservation. This avoids creating reservations you'll immediately release:

```python
import os
import uuid

from runcycles import (
    Action, Amount, CyclesClient, CyclesConfig, DecisionRequest, Subject, Unit,
)

client = CyclesClient(CyclesConfig(
    base_url=os.environ["CYCLES_BASE_URL"],
    api_key=os.environ["CYCLES_API_KEY"],
))

def process_document(doc, doc_id):
    for attempt in range(10):
        resp = client.decide(DecisionRequest(
            idempotency_key=f"doc-{doc_id}-decide-{uuid.uuid4()}",
            subject=Subject(tenant="acme-corp", workflow=f"doc-{doc_id}"),
            action=Action(kind="llm.completion", name="gpt-4o"),
            estimate=Amount(amount=5_000_000, unit=Unit.USD_MICROCENTS),
        ))
        if resp.body["decision"] == "DENY":
            return f"Document {doc_id}: budget exhausted after {attempt} attempts."
        response = call_llm_guarded(f"Process this document: {doc}")
        if validate(response):
            return response
```

### 3. Track retry cost separately

Use the `metrics` field on commit to tag retries. Inside a `@cycles`-guarded function, `get_cycles_context()` exposes the live reservation context; setting `ctx.metrics` attaches custom metrics to the commit. This lets you build dashboards that show what fraction of your spend goes to retries versus first attempts:

```python
from runcycles import CyclesMetrics, cycles, get_cycles_context

@cycles(
    estimate=5_000_000,
    action_kind="llm.completion",
    action_name="gpt-4o",
)
def call_llm_guarded(prompt: str, attempt: int = 0) -> str:
    result = call_llm(prompt)
    ctx = get_cycles_context()
    ctx.metrics = CyclesMetrics(
        custom={"retry_attempt": attempt, "is_retry": attempt > 0},
    )
    return result
```

### 4. Set a maximum retry budget as a fraction of first-attempt cost

A useful heuristic: retries should never cost more than 2x the original call. If your first attempt costs $0.05, cap total retry spend at $0.10. This prevents the long tail of expensive retries:

```python
MAX_RETRY_MULTIPLIER = 2
first_attempt_cost = 5_000_000  # microcents

def process_with_capped_retries(doc, doc_id):
    total_spent = 0
    max_retry_budget = first_attempt_cost * MAX_RETRY_MULTIPLIER
    for attempt in range(10):
        if attempt > 0 and total_spent >= max_retry_budget:
            return f"Document {doc_id}: retry budget exhausted."
        response = call_llm_guarded(f"Process this document: {doc}")
        # @cycles commits the estimate unless an actual-cost expression is
        # configured, so the same constant tracks spend here. For the live
        # reservation details (reservation_id, caps, balances), call
        # get_cycles_context() inside the guarded function.
        total_spent += first_attempt_cost
        if validate(response):
            return response
```

### 5. Use circuit breakers for persistent failures

If multiple documents in a batch hit max retries, the issue is likely systemic (model degradation, bad prompt template). A circuit breaker stops the entire batch early:

```python
class RetryCircuitBreaker:
    def __init__(self, threshold=5):
        self.failure_count = 0
        self.threshold = threshold

    def record_exhausted_retries(self):
        self.failure_count += 1
        if self.failure_count >= self.threshold:
            raise SystemError(
                f"{self.failure_count} documents exhausted retries. "
                "Halting batch — likely systemic issue."
            )
```

## Next steps

- [Idempotency, Retries and Concurrency](/concepts/idempotency-retries-and-concurrency-why-cycles-is-built-for-real-failure-modes) — how Cycles handles retries safely
- [Error Codes and Error Handling](/protocol/error-codes-and-error-handling-in-cycles) — understanding IDEMPOTENCY_MISMATCH
- [Runaway Agents and Tool Loops](/incidents/runaway-agents-tool-loops-and-budget-overruns-the-incidents-cycles-is-designed-to-prevent) — the broader runaway agent problem
- [AI Agent Cost Management: The Complete Guide](/blog/ai-agent-cost-management-guide) — how monitoring and alerting tiers handle retry storms before enforcement



# Runaway Agents, Tool Loops, and Budget Overruns: The Incidents Cycles Is Designed to Prevent

Most infrastructure gets adopted after a painful incident.

Not because the idea was unclear.  
Because the failure became expensive enough to matter.

Cycles exists for a specific class of incidents:

- runaway agent execution
- recursive tool loops
- retry storms that multiply spend
- background workflows that drift out of bounds
- tenant over-consumption
- side-effecting systems that keep acting longer than intended

These are not hypothetical problems.

They are what happens when autonomous systems are allowed to keep executing without a clear runtime authority.

## The common pattern behind these incidents

The root problem is usually not one bad model call.

It is uncontrolled accumulation over time.

A system begins with a reasonable action:

- answer a question
- call a model
- retrieve context
- invoke a tool
- retry a failed step
- continue a background workflow

Then one of several things happens:

- it loops
- it retries repeatedly
- it fans out across tools
- it recurses into additional steps
- it continues after the initiating request is gone
- it stays within request limits but exceeds intended total spend

The incident is rarely obvious at the start.

It becomes obvious only after enough exposure has already accumulated.

That is exactly the gap Cycles is designed to close.

## Incident type 1: Runaway agent loops

This is one of the clearest failure modes.

An agent is given a task.  
It plans.  
It reasons.  
It calls a tool.  
It reevaluates.  
It calls again.  
Then again.

Each step may look individually valid.

The problem is the total chain.

The agent may stay functionally “alive” long after it has stopped being useful.

### What makes this dangerous

- cost grows with each iteration
- the loop may not violate request-per-second limits
- the workflow may appear healthy from a latency perspective
- the failure is often logical, not infrastructural

By the time someone notices, the system has already consumed real budget.

### What Cycles changes

Cycles introduces a run-level budget boundary.

That means a run can be given a bounded execution envelope before it starts.

If the run exhausts that envelope, the system can:

- stop
- degrade
- switch to a smaller model
- disable expensive tools
- exit gracefully

Instead of hoping the loop ends on its own, the platform enforces that it cannot continue indefinitely.

## Incident type 2: Recursive tool loops

Many agent systems now use tools as part of normal execution.

That is powerful, but it also creates a new failure surface.

A tool call may trigger:

- another model step
- another tool selection
- another retrieval pass
- another external API call

Sometimes this is intentional.  
Sometimes it becomes accidental recursion.

### A common failure shape

An agent tries to achieve a task by alternating between planning and tool invocation.

The tool result is incomplete or ambiguous.  
The model decides to try again.  
The same or similar tool path repeats.

This may not look like a classic software infinite loop.  
It may look like a sequence of plausible, locally valid decisions.

But operationally, the effect is similar.

### What Cycles changes

Cycles allows tool-calling paths to operate inside bounded budgets.

That means expensive or risky tools do not merely rely on agent judgment. They also rely on budget availability.

If a recursive chain keeps consuming exposure, it can hit a hard ceiling before becoming an open-ended incident.

## Incident type 3: Retry storms that multiply spend

Retries are necessary.

They are also dangerous when execution is expensive.

A transient error occurs.  
The system retries.  
Then retries again.  
Then downstream components retry as well.

Each retry may appear operationally justified.

But collectively they can produce:

- duplicate model usage
- repeated external API charges
- repeated side effects
- budget consumption far above the original intent

### Why this is tricky

Retry behavior often emerges across layers:

- client retries
- worker retries
- message redelivery
- provider-level transient failures
- workflow-level retry policies

A team may believe it has only one retry path when in reality several are active at once.

### What Cycles changes

Cycles is built around reservation, commit, release, and retry-safe lifecycle handling.

That creates a stronger basis for budget control under repeated attempts.

Instead of treating every retry as disconnected spend, the runtime can reason about bounded execution more intentionally.

The goal is not to remove retries.

The goal is to prevent retries from silently becoming budget explosions.

## Incident type 4: Background workflows that drift out of bounds

Many systems start with synchronous user-triggered actions.

Then they evolve.

Work moves into background jobs, queue consumers, autonomous workflows, scheduled agents, and multi-step processing pipelines.

At that point, the original user request may be gone while the system is still acting.

### Why this matters

Once work becomes long-lived or asynchronous, teams lose the natural boundary of a single request-response cycle.

That means the system may continue to:

- call models
- invoke tools
- write state
- trigger follow-up jobs
- accumulate cost

without a clean execution envelope.

### What Cycles changes

Cycles gives background execution a budget boundary.

A workflow or run can reserve bounded room to act before it continues, even if it is no longer tied to an active foreground request.

That makes asynchronous autonomy more governable.

## Incident type 5: Tenant over-consumption

In multi-tenant systems, not every incident is caused by a single bad run.

Sometimes the issue is aggregate consumption.

One tenant may:

- use a feature far more heavily than expected
- trigger many concurrent runs
- repeatedly invoke expensive workflows
- consume shared capacity beyond its intended share

Without a strong budget model, teams often discover this through:

- a provider bill
- degraded shared performance
- surprise usage spikes
- unhappy other tenants

### Why rate limits are not enough

A tenant can remain within request velocity constraints and still exceed intended total exposure over time.

This is especially true for long-running or autonomous workloads.

### What Cycles changes

Cycles supports tenant-level budgets as part of hierarchical governance.

That means every action can be checked not only against local run or workflow limits, but also against broader tenant boundaries.

This turns tenant isolation from post-hoc analytics into pre-execution control.

## Incident type 6: Side-effecting systems that continue too long

Some autonomous systems do more than think.

They act.

They may:

- send emails
- create tickets
- write to databases
- trigger payments
- call downstream business systems
- initiate deployments

At that point, the incident is not just cloud spend.  
It is operational side effect.

### Why this is more serious

A long-running reasoning loop is costly.  
A long-running side-effect loop can be destructive.

The platform may need to distinguish between:

- low-risk model inference
- medium-risk retrieval
- high-risk external action

### What Cycles changes

Cycles allows these actions to be governed as budgeted exposure, not just as traffic.

That means the system can decide whether an action is still allowed to proceed under the current budget, scope, and policy state.

This is how “autonomous execution” becomes something operators can actually bound.

## The operational theme behind all of these incidents

All of these incidents share the same deeper issue:

**the system keeps acting after it should have stopped.**

Not necessarily because it is malicious.  
Not necessarily because it is broken in the classic sense.

But because nothing in the runtime enforces a bounded execution envelope.

That is the problem Cycles is designed to solve.

## What teams often use instead

Before adopting a runtime authority, teams usually piece together partial controls such as:

- provider dashboards
- usage alerts
- request rate limits
- hardcoded loop counters
- timeout tuning
- kill switches
- tenant usage reports
- manual intervention

These controls are often useful, but fragmented.

They usually fail in one of two ways:

- they react too late
- they are too coarse to map cleanly onto autonomous execution

Cycles is not trying to remove all of these tools.

It is trying to add the missing control layer that turns bounded execution into a runtime property.

## What prevention looks like in practice

Cycles does not prevent incidents by “observing harder.”

It prevents them by changing the execution model.

At a high level:

1. an action declares intended exposure
2. budget is reserved before work proceeds
3. execution happens within that bounded envelope
4. actual usage is committed
5. unused remainder is released
6. further work is denied or degraded when budgets are exhausted

That changes the system from:

::: info
keep going until something external notices
:::

to:

::: info
continue only while bounded execution is still authorized
:::

That is a different operating model.

## See it in action: the runaway agent demo

The [Runaway Agent Demo](/demos/) demonstrates exactly this failure mode with a runnable example. No LLM API key required.

The scenario: a customer support bot drafts a response, evaluates its quality, and refines it in a loop until the quality score exceeds 8.0. The bug is that the quality evaluator never returns above 6.9. Without a budget boundary, the agent loops indefinitely.

The demo runs the same agent twice:

1. **Without Cycles** — the agent runs for 30 seconds, making ~600 calls and spending ~$6.00 before being auto-terminated. In production, there would be no auto-termination.
2. **With Cycles (budget: $1.00)** — the agent hits the budget ceiling after ~100 calls. The Cycles server returns `409 BUDGET_EXCEEDED`, the `@cycles` decorator raises `BudgetExceededError`, and the agent stops cleanly.

The entire integration diff between the unguarded and guarded versions is three `@cycles` decorators and one `except BudgetExceededError` block.

## Why this matters now

As AI systems become more autonomous, incidents are shifting.

The old failure model was often:

- one bad request
- one high-latency call
- one failed dependency

The new failure model is often:

- too much valid work
- repeated steps that stay locally reasonable
- distributed retries
- side effects that accumulate
- autonomy that continues past useful bounds

This is exactly why autonomous software needs more than traffic shaping and dashboards.

It needs runtime authority.

## Summary

Cycles is designed to prevent incidents such as:

- runaway agent loops
- recursive tool chains
- retry-driven budget explosions
- background workflows drifting out of bounds
- tenant over-consumption
- excessive or repeated side effects

These incidents all come from the same core gap:

the absence of a runtime control layer that can bound autonomous execution before cost and side effects accumulate too far.

That is what Cycles provides.

It gives teams a way to move from:

- hoping systems stay within acceptable bounds

to:

- enforcing bounded execution intentionally

## Next steps

To explore the Cycles stack:

- Try the [Demos](/demos/) — runaway agent (cost control) and action authority (action blocking) scenarios
- Read the [Cycles Protocol](https://github.com/runcycles/cycles-protocol)
- Run the [Cycles Server](https://github.com/runcycles/cycles-server)
- Manage budgets with [Cycles Admin](https://github.com/runcycles/cycles-server-admin)
- Integrate with Python using the [Python Client](/quickstart/getting-started-with-the-python-client)
- Integrate with TypeScript using the [TypeScript Client](/quickstart/getting-started-with-the-typescript-client)
- Integrate with Spring Boot or Spring AI using the [Spring Boot starter](https://github.com/runcycles/cycles-spring-boot-starter) or the [Spring AI starter](https://github.com/runcycles/cycles-spring-ai-starter)
- [The True Cost of Uncontrolled AI Agents](/blog/true-cost-of-uncontrolled-agents) — real-world costs and failure modes of agents without budget limits
- [5 Real-World AI Agent Failures That Budget Controls Would Have Prevented](/blog/ai-agent-failures-budget-controls-prevent) — concrete failure scenarios and what each tier prevents



# Scope Misconfiguration and Budget Leaks

A failure mode where budget is consumed from unintended scopes due to misconfigured subject fields, or where budget is not properly tracked because scopes don't match.

## The incident

A team sets up per-workspace budgets but their application doesn't consistently pass the `workspace` field in reservations. Some calls include `workspace=prod`, others omit it entirely. The result: calls without a workspace field only check the tenant-level budget, bypassing the workspace limit entirely.

### Example

**Budget setup:**

```
tenant:acme-corp                    → $100/month
tenant:acme-corp/workspace:prod     → $50/month
```

**Application code (inconsistent):**

```python
# Route A: Includes workspace — checks both scopes
@cycles(estimate=2000000, action_kind="llm.completion",
        action_name="gpt-4o", workspace="prod")
def route_a(prompt):
    ...

# Route B: Missing workspace — only checks tenant scope
@cycles(estimate=2000000, action_kind="llm.completion",
        action_name="gpt-4o")
def route_b(prompt):
    ...
```

Route B spends against `tenant:acme-corp` but never touches `tenant:acme-corp/workspace:prod`. The workspace budget appears underutilized while the tenant budget drains from both routes.

## Why this matters

- **Budget bypass.** If the workspace budget is meant to limit production spend, calls that skip the workspace field are unaccounted for at that level.
- **Misleading balances.** The workspace balance report shows less spending than actually occurred. Operators think production is within limits, but the tenant-level budget tells a different story.
- **No enforcement gap.** Cycles enforces exactly what it's told. If the subject doesn't include a scope level, that level is not checked.

## Severity and impact

Scope misconfiguration is uniquely dangerous because it is **silent**. Unlike a budget exceeded error or a denied reservation, a misconfigured scope produces no errors. Calls succeed, money is spent, and the budget reports look normal — until you realize the per-workspace limits you carefully configured are being bypassed entirely.

**Budget bypass scenario:**

```
Budget setup:
  tenant:acme-corp                    → $100/month
  tenant:acme-corp/workspace:prod     → $50/month

Route A (correct scope):   50 calls × $0.50 = $25 → charged to both tenant and workspace
Route B (missing workspace): 200 calls × $0.50 = $100 → charged to tenant only

Result:
  tenant:acme-corp        → $125 spent (OVER BUDGET)
  workspace:prod           → $25 spent  (looks fine!)
```

The workspace dashboard shows $25 spent — well within the $50 limit. But the tenant is $25 over budget because Route B bypassed workspace-level enforcement entirely. An operator looking at workspace reports sees no problem.

**Cascading misconfiguration.** When one team gets scope construction wrong, other teams sharing the same tenant scope bear the cost. Team A's misconfigured calls drain the tenant budget, causing Team B's correctly-scoped calls to be denied with `BUDGET_EXCEEDED` at the tenant level even though their workspace budget has room.

**Audit failure.** Scope mismatches break cost attribution. If finance needs to know how much the "prod" workspace spent, the answer is incomplete because Route B's spend is invisible at that scope level. This makes chargebacks and cost allocation unreliable.

## Common misconfiguration patterns

### 1. Inconsistent subject fields across routes

Different code paths construct subjects differently. One team uses `workspace`, another doesn't.

**Fix:** Centralize subject construction:

```python
def build_subject(**overrides):
    return {
        "tenant": os.environ["CYCLES_TENANT"],
        "workspace": os.environ.get("CYCLES_WORKSPACE", "default"),
        **overrides,
    }
```

### 2. Missing budget at intermediate scope levels

Budgets exist at `tenant:acme` and `tenant:acme/workspace:prod/app:chatbot`, but not at `tenant:acme/workspace:prod`. Per the protocol's skip semantics, scopes without a budget are **skipped** — the reservation checks and debits only the levels that have budgets, so the missing middle level is silently unenforced (no error is raised). Only if *none* of the affected scopes has a budget does the reservation fail, with `404 NOT_FOUND` ("Budget not found for provided scope"). The risk here is not a spurious denial — it's a level you believe is capped that isn't.

**Fix:** Create budgets at every scope level that appears in your subject hierarchy:

```bash
# Create budget at every level
curl -s -X POST .../budgets -d '{"scope": "tenant:acme"}'
curl -s -X POST .../budgets -d '{"scope": "tenant:acme/workspace:prod"}'
curl -s -X POST .../budgets -d '{"scope": "tenant:acme/workspace:prod/app:chatbot"}'
```

### 3. Wrong scope order

Cycles scopes follow a fixed hierarchy: `tenant → workspace → app → workflow → agent → toolset`. Providing fields in a different conceptual mapping (e.g., using `agent` for what's really a workspace concept) causes budget checks against the wrong ledgers.

**Fix:** Map your domain concepts to Cycles scopes consistently. See [Scope Derivation](/protocol/how-scope-derivation-works-in-cycles).

### 4. Typos in scope values

`workspace: "prod"` vs `workspace: "production"` creates two separate scope paths with separate budgets. One gets all the traffic, the other sits unused.

**Fix:** Use constants or enums for scope values, not string literals.

### 5. Dynamic scope values from user input

When scope values are derived from user input (API parameters, form fields, URL paths), unsanitized values create unpredictable scope paths:

```python
# DANGEROUS: user-controlled scope value
@cycles(estimate=2000000, action_kind="llm.completion",
        action_name="gpt-4o",
        workspace=request.headers.get("X-Workspace"))
def handle_request(prompt):
    ...
```

If a user sends `X-Workspace: prod/agent:attacker`, the scope path becomes `tenant:acme-corp/workspace:prod/agent:attacker` — an injected level that has no budget configured. Under the protocol's skip semantics, budget-less scopes are simply skipped: the call is enforced only against whatever parent budgets exist, the injected level goes unenforced, and cost attribution at that level is polluted. The call would fail (`404 NOT_FOUND`) only in the unlikely case that *no* scope in the path has a budget.

**Fix:** Validate and sanitize scope values against an allowlist:

```python
VALID_WORKSPACES = {"prod", "staging", "dev"}

def safe_workspace(raw_value: str) -> str:
    sanitized = raw_value.strip().lower()
    if sanitized not in VALID_WORKSPACES:
        raise ValueError(f"Invalid workspace: {raw_value}")
    return sanitized

@cycles(estimate=2000000, action_kind="llm.completion",
        action_name="gpt-4o",
        workspace=safe_workspace(request.headers.get("X-Workspace", "default")))
def handle_request(prompt):
    ...
```

In TypeScript:

```typescript
const VALID_WORKSPACES = new Set(["prod", "staging", "dev"]);

function safeWorkspace(raw: string | undefined): string {
  const sanitized = (raw ?? "default").trim().toLowerCase();
  if (!VALID_WORKSPACES.has(sanitized)) {
    throw new Error(`Invalid workspace: ${raw}`);
  }
  return sanitized;
}
```

## Detection

### Check for scope inconsistency

Compare the scopes that have budget with the scopes appearing in reservation activity:

```bash
# Budget scopes
curl -s "http://localhost:7878/v1/balances?tenant=acme-corp" \
  -H "X-Cycles-API-Key: $API_KEY" | jq '.balances[].scope'

# Active reservations show which scopes are being used
curl -s "http://localhost:7878/v1/reservations?tenant=acme-corp&status=ACTIVE" \
  -H "X-Cycles-API-Key: $API_KEY" | jq '.reservations[].subject'
```

If reservations are hitting scopes that don't appear in your budget list, you have a configuration gap.

### Use dry-run mode to audit

Run in [shadow mode](/how-to/shadow-mode-in-cycles-how-to-roll-out-budget-enforcement-without-breaking-production) to see all the scopes your application actually uses before creating budgets.

### TypeScript detection example

You can programmatically detect scope gaps by comparing budget scopes against reservation scopes:

```typescript
import { CyclesClient, CyclesConfig } from "runcycles";

const client = new CyclesClient(new CyclesConfig({
  baseUrl: process.env.CYCLES_BASE_URL!,
  apiKey: process.env.CYCLES_API_KEY!,
}));

async function detectScopeGaps(tenant: string): Promise<string[]> {
  const balancesResp = await client.getBalances({ tenant });
  const reservationsResp = await client.listReservations({ tenant, status: "ACTIVE" });

  const balances = balancesResp.body!.balances as Array<{ scope_path: string }>;
  const reservations = reservationsResp.body!.reservations as Array<{ scope_path: string }>;

  const budgetScopes = new Set(balances.map((b) => b.scope_path));

  const gaps = new Set<string>();
  for (const r of reservations) {
    if (!budgetScopes.has(r.scope_path)) {
      gaps.add(r.scope_path);
    }
  }
  return [...gaps];
}

// Usage — deepest reservation scopes with no budget at that level (skipped, unenforced)
const gaps = await detectScopeGaps("acme-corp");
if (gaps.length > 0) {
  console.warn("Reservations whose deepest scope has no budget (level unenforced):", gaps);
}
```

## Monitoring

### Alerting for scope mismatches

::: warning Planned metrics — requires balance-polling sidecar
`cycles_reservations_created_total{scope=...}` and `cycles_scope_spent_total{level=...}` are on the roadmap but not emitted by the current server builds (runtime 0.1.25.58). The server does register `cycles_*` operation counters (`cycles_reservations_*`, `cycles_events_total`, `cycles_overdraft_incurred_total`), but their tags are `decision` / `reason` / `overage_policy` / `tenant` — there is no scope label on them, nor on `http_server_requests_seconds*` — so scope-aware alerts require a sidecar that polls `GET /v1/balances` and `GET /v1/reservations` and pushes labelled gauges (`cycles_scope_spent`, `cycles_scope_allocated`, `cycles_reservations_created`) into your metrics pipeline. See [Balance-polling alerts](/how-to/monitoring-and-alerting#balance-polling-alerts-for-signals-without-a-counter). Once those gauges exist, the rules below apply as-is.
:::

```yaml
# Alert when reservations hit scopes that have no configured budget.
# Requires sidecar gauges: cycles_reservations_created{scope=...}, cycles_scope_allocated{scope=...}.
- alert: CyclesScopeWithoutBudget
  expr: |
    cycles_reservations_created{scope=~".+"}
    unless on(scope) cycles_scope_allocated
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Reservations hitting scope {{ $labels.scope }} which has no budget"

# Alert when a scope's spend diverges significantly from its child scopes
# (indicates traffic bypassing child scope). Requires sidecar gauges with a level label.
- alert: CyclesScopeSpendMismatch
  expr: |
    cycles_scope_spent{level="tenant"}
    - sum(cycles_scope_spent{level="workspace"}) by (tenant)
    > 1000000
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Tenant spend exceeds sum of workspace spend — possible scope bypass"

# Alert when a workspace scope shows zero spend while tenant scope is active.
- alert: CyclesInactiveChildScope
  expr: |
    cycles_scope_spent{level="workspace"} == 0
    and on(tenant) cycles_scope_spent{level="tenant"} > 0
  for: 30m
  labels:
    severity: info
  annotations:
    summary: "Workspace {{ $labels.workspace }} has zero spend — check for scope misconfiguration"
```

### Key metrics to track

- **Budget coverage ratio:** scopes with budgets vs distinct scopes in reservations. Should be 1.0.
- **Parent-child spend delta:** difference between parent spend and sum of child spend. Non-zero means traffic is bypassing child scopes.
- **Distinct scope count over time:** sudden increases suggest dynamic scope values from user input (pattern #5).

For detailed monitoring setup, see [Monitoring and Alerting](/how-to/monitoring-and-alerting).

## Testing scope configuration

### Python: verify all routes include required scope fields

```python
import pytest
from unittest.mock import patch
from runcycles import get_cycles_context

REQUIRED_SCOPE_LEVELS = {"tenant", "workspace"}

def _capture_scope_path(captured):
    """Mock the LLM call and grab the reservation context while it is live.

    get_cycles_context() only returns the reservation context inside the
    @cycles-guarded call, so capture scope_path from within the mock.
    """
    def _mock(prompt):
        ctx = get_cycles_context()
        captured["scope_path"] = ctx.scope_path
        return "mocked"
    return _mock

def test_route_a_includes_all_scopes():
    """Verify that route_a's reservation covers all required scope levels."""
    captured = {}
    with patch("myapp.call_llm", side_effect=_capture_scope_path(captured)):
        route_a("test prompt")

    missing = {
        level for level in REQUIRED_SCOPE_LEVELS
        if f"{level}:" not in captured["scope_path"]
    }
    assert not missing, f"Route A missing scope levels: {missing}"

def test_route_b_includes_all_scopes():
    """Verify that route_b's reservation covers all required scope levels."""
    captured = {}
    with patch("myapp.call_llm", side_effect=_capture_scope_path(captured)):
        route_b("test prompt")

    missing = {
        level for level in REQUIRED_SCOPE_LEVELS
        if f"{level}:" not in captured["scope_path"]
    }
    assert not missing, f"Route B missing scope levels: {missing}"
```

### TypeScript: centralized scope builder with tests

```typescript
import { withCycles } from "runcycles";

// Centralized scope builder — all routes use this
interface ScopeConfig {
  tenant: string;
  workspace: string;
  app?: string;
}

function buildScope(): ScopeConfig {
  const tenant = process.env.CYCLES_TENANT;
  const workspace = process.env.CYCLES_WORKSPACE;
  if (!tenant) throw new Error("CYCLES_TENANT is required");
  if (!workspace) throw new Error("CYCLES_WORKSPACE is required");
  return { tenant, workspace };
}

// Test that buildScope rejects missing fields
describe("buildScope", () => {
  it("throws if CYCLES_TENANT is missing", () => {
    delete process.env.CYCLES_TENANT;
    process.env.CYCLES_WORKSPACE = "prod";
    expect(() => buildScope()).toThrow("CYCLES_TENANT is required");
  });

  it("throws if CYCLES_WORKSPACE is missing", () => {
    process.env.CYCLES_TENANT = "acme-corp";
    delete process.env.CYCLES_WORKSPACE;
    expect(() => buildScope()).toThrow("CYCLES_WORKSPACE is required");
  });

  it("returns all required fields", () => {
    process.env.CYCLES_TENANT = "acme-corp";
    process.env.CYCLES_WORKSPACE = "prod";
    const scope = buildScope();
    expect(scope).toHaveProperty("tenant", "acme-corp");
    expect(scope).toHaveProperty("workspace", "prod");
  });
});
```

For more testing patterns, see [Testing with Cycles](/how-to/testing-with-cycles).

## Prevention

1. **Centralize subject construction.** Don't let individual routes build subjects ad hoc.
2. **Use environment variables for common fields.** Tenant, workspace, and app should come from configuration, not hardcoded strings.
3. **Audit scope usage regularly.** Compare active reservation scopes against budget scopes.
4. **Create budgets at all hierarchy levels.** Any scope that appears in a subject needs a budget.
5. **Use shadow mode when adding new scope levels.** Verify the new scopes match before enforcing.

## Next steps

- [Scope Derivation](/protocol/how-scope-derivation-works-in-cycles) — how Cycles builds scope paths from subject fields
- [Budget Allocation and Management](/how-to/budget-allocation-and-management-in-cycles) — creating and funding budgets
- [Shadow Mode Rollout](/how-to/shadow-mode-in-cycles-how-to-roll-out-budget-enforcement-without-breaking-production) — testing scopes without enforcement
- [AI Agent Budget Patterns: A Practical Guide](/blog/agent-budget-patterns-visual-guide) — six common patterns to avoid scope misconfiguration

---

# Configuration



# Client Configuration Reference for the Cycles Spring Boot Starter

::: tip Using Python?
See the [Python Client Configuration Reference](/configuration/python-client-configuration-reference) instead.
:::

This is the complete reference for all configuration properties available in the Cycles Spring Boot Starter.

::: tip Using Spring AI advisors?
This page covers the underlying `cycles.*` client properties. See the [Spring AI Starter Configuration Reference](/configuration/spring-ai-starter-configuration-reference) for the separate `cycles.spring-ai.*` advisor, token-estimation, tool-gating, and tracing properties.
:::

All properties are under the `cycles` prefix in your project's `application.yml` (or `application.properties`).

## Required properties

| Property | Type | Description |
|---|---|---|
| `cycles.base-url` | String | Base URL of the Cycles server (e.g., `http://localhost:7878`) |
| `cycles.api-key` | String | API key for authentication |

If either is missing or blank, the application will fail to start with a configuration error.

## Subject defaults

These properties set default values for the Subject fields used in `@Cycles` annotations. They apply to all annotated methods unless overridden at the annotation level or by a `CyclesFieldResolver`.

| Property | Type | Default | Description |
|---|---|---|---|
| `cycles.tenant` | String | (none) | Default tenant |
| `cycles.workspace` | String | (none) | Default workspace |
| `cycles.app` | String | (none) | Default application name |
| `cycles.workflow` | String | (none) | Default workflow |
| `cycles.agent` | String | (none) | Default agent |
| `cycles.toolset` | String | (none) | Default toolset |

### Resolution order

For each Subject field, the starter resolves the value using this priority:

1. **Annotation attribute** — if set on the `@Cycles` annotation, it wins
2. **Configuration property** — if set in `application.yml`
3. **CyclesFieldResolver bean** — if a bean named after the field exists (e.g., a bean named `"tenant"` implementing `CyclesFieldResolver`)

If none of these provide a value, the field is omitted from the request.

### Per-annotation overrides and budget scope targeting

When you override a subject field on `@Cycles`, the resolved subject changes, which targets a different budget scope on the server.

**Example:** Given this configuration:

```yaml
cycles:
  tenant: acme
  workspace: production
  app: support-bot
```

A method with `@Cycles(value = "1000", workspace = "staging")` resolves to:

| Field | Source | Resolved value |
|---|---|---|
| `tenant` | config | `acme` |
| `workspace` | **annotation** | `staging` |
| `app` | config | `support-bot` |

The reservation targets scope `tenant:acme/workspace:staging/app:support-bot` instead of `tenant:acme/workspace:production/app:support-bot`.

**Budget scope implications:** Each level in the scope hierarchy has an independent budget. If you have:
- `tenant:acme/workspace:production` → $10,000 budget
- `tenant:acme/workspace:staging` → $1,000 budget

Using `workspace = "staging"` on an annotation targets the $1,000 staging budget. Without the override, the same method targets the $10,000 production budget.

::: warning
A reservation must pass budget checks at **all** affected scope levels. If you set `tenant` and `workspace`, the server checks remaining budget at both `tenant:acme` and `tenant:acme/workspace:staging`. If either scope is exhausted, the reservation is denied.
:::

## `@Cycles` annotation attributes

The `@Cycles` annotation controls reservation behavior per-method. These are separate from the `application.yml` configuration above. For full documentation and examples, see [Getting Started with the Spring Boot Starter — Annotation attributes](/quickstart/getting-started-with-the-cycles-spring-boot-starter#annotation-attributes).

| Attribute | Type | Default | Description |
|---|---|---|---|
| `value` | `String` | `""` | SpEL expression for estimated cost. Shorthand: `@Cycles("1000")`. Synonym for `estimate`. |
| `estimate` | `String` | `""` | SpEL expression for estimated cost. Synonym for `value`. |
| `actual` | `String` | `""` | SpEL expression for actual cost, evaluated after method returns. `#result` is bound to the return value. |
| `metadata` | `String` | `""` | SpEL expression for commit metadata, evaluated after method returns (`#result` available). Must yield `Map<String, Object>`. Merged with programmatic `CyclesContextHolder` metadata; programmatic wins on key conflicts. Since 0.2.5. |
| `actionKind` | `String` | `""` | Action category (e.g. `"llm.completion"`). Defaults to declaring class simple name if blank. |
| `actionName` | `String` | `""` | Action identifier (e.g. `"gpt-4"`). Defaults to method name if blank. |
| `actionTags` | `String[]` | `{}` | Tags for filtering and reporting (e.g. `{"prod", "customer-facing"}`). |
| `unit` | `String` | `"USD_MICROCENTS"` | Budget unit: `USD_MICROCENTS`, `TOKENS`, `CREDITS`, `RISK_POINTS`. |
| `ttlMs` | `long` | `60000` | Reservation TTL in milliseconds (1,000–86,400,000). |
| `gracePeriodMs` | `long` | `-1` | Grace period after TTL expiry in milliseconds. When `-1`, the server applies its default (5000ms). Valid range: 0–60,000. |
| `overagePolicy` | `String` | `"ALLOW_IF_AVAILABLE"` | `"REJECT"`, `"ALLOW_IF_AVAILABLE"`, or `"ALLOW_WITH_OVERDRAFT"`. |
| `dryRun` | `boolean` | `false` | Shadow-mode evaluation. If `true`, server evaluates without persisting; guarded method does NOT execute. |
| `useEstimateIfActualNotProvided` | `boolean` | `true` | When `true` and `actual` is blank, use the estimate as actual at commit time. |
| `tenant` | `String` | `""` | Subject tenant override (takes precedence over config and resolver). |
| `workspace` | `String` | `""` | Subject workspace override. |
| `app` | `String` | `""` | Subject app override. |
| `workflow` | `String` | `""` | Subject workflow override. |
| `agent` | `String` | `""` | Subject agent override. |
| `toolset` | `String` | `""` | Subject toolset override. |
| `dimensions` | `String[]` | `{}` | Custom dimensions as `"key=value"` pairs (e.g. `{"cost_center=engineering"}`). |

::: tip SpEL on subject attributes (since 0.2.1)
Subject attributes (`tenant`, `workspace`, `app`, `workflow`, `agent`, `toolset`) whose value starts with `#` are evaluated as SpEL against the method invocation, e.g. `tenant = "#tenantId"`. Literal values are passed through unchanged. Parse or evaluation failures surface as `ParseException`/`SpelEvaluationException` when the aspect runs, before the reservation is created.
:::

## HTTP configuration

| Property | Type | Default | Description |
|---|---|---|---|
| `cycles.http.connect-timeout` | Duration | `2s` | TCP connection timeout to the Cycles server |
| `cycles.http.read-timeout` | Duration | `5s` | Read timeout for responses from the Cycles server |

Duration values use Spring Boot duration syntax: `2s`, `500ms`, `1m`, etc.

### Example

```yaml
cycles:
  http:
    connect-timeout: 3s
    read-timeout: 10s
```

For long-running operations where the server may take longer to respond (e.g., under heavy load), increase the read timeout.

For automatic heartbeat safety, keep the combined connect and read timeout well below half the smallest expected reservation lease. The heartbeat reserves two complete attempt budgets plus a safety margin; a 30-second attempt budget cannot establish a positive cadence inside the default 60-second TTL.

## Retry configuration

Controls the durable settlement retry engine.

| Property | Type | Default | Description |
|---|---|---|---|
| `cycles.retry.enabled` | boolean | `true` | Enable automatic commit retries |
| `cycles.retry.max-attempts` | int | `5` | Maximum number of retry attempts |
| `cycles.retry.initial-delay` | Duration | `500ms` | Delay before the first retry |
| `cycles.retry.multiplier` | double | `2.0` | Backoff multiplier between retries |
| `cycles.retry.max-delay` | Duration | `30s` | Maximum delay between retries |
| `cycles.retry.flush-timeout` | Duration | `10s` | Bounded wait for in-flight retries during Spring context shutdown |

### Journal configuration

| Property | Type | Default | Description |
|---|---|---|---|
| `cycles.journal.enabled` | boolean | `true` | Persist unresolved known-actual settlement across JVM restarts |
| `cycles.journal.dir` | String | (none) | Journal base directory; unset uses `~/.runcycles/commit-journal` |

### How retry works

Known actual usage is journaled before the first commit request. When settlement fails transiently, the engine schedules a same-key retry using exponential backoff:

```
Attempt 1: wait 500ms
Attempt 2: wait 1000ms
Attempt 3: wait 2000ms
Attempt 4: wait 4000ms
Attempt 5: wait 8000ms (capped at max-delay)
```

HTTP 429 honors a valid `Retry-After` floor and persists it across restart. Authentication failures and unclassifiable 4xx responses stop the current run but retain the record. A genuine, understood rejection removes it.

Only a schema-valid HTTP `200` commit or schema-valid HTTP `201` event proves success. If commit returns HTTP 410 or `RESERVATION_EXPIRED`, the journal switches to event mode before the starter calls `POST /v1/events` with the original idempotency key.

The journal is partitioned by server and principal. Configure `cycles.tenant` so pending records remain discoverable after API-key rotation. Records do not store API keys, but they contain settlement bodies and metadata; protect the directory as sensitive application state.

### Disabling retry

```yaml
cycles:
  retry:
    enabled: false
```

This disables active retries, not journaling. Failed known-actual settlement remains on disk while `cycles.journal.enabled=true`. Disable both only when the application supplies equivalent durable recovery.

### Aggressive retry for critical commits

```yaml
cycles:
  retry:
    max-attempts: 10
    initial-delay: 200ms
    multiplier: 1.5
    max-delay: 60s
```

## Full configuration example

Add the following to your project's `application.yml`:

```yaml
cycles:
  # Required
  base-url: ${CYCLES_BASE_URL:http://localhost:7878}
  api-key: ${CYCLES_API_KEY}

  # Subject defaults
  tenant: acme
  workspace: production
  app: support-bot

  # HTTP settings
  http:
    connect-timeout: 2s
    read-timeout: 5s

  # Commit retry
  retry:
    enabled: true
    max-attempts: 5
    initial-delay: 500ms
    multiplier: 2.0
    max-delay: 30s
    flush-timeout: 10s

  # Durable settlement journal
  journal:
    enabled: true
    # dir: /var/lib/my-app/cycles-commit-journal
```

## Equivalent application.properties

Alternatively, add to your project's `application.properties`:

```properties
cycles.base-url=${CYCLES_BASE_URL:http://localhost:7878}
cycles.api-key=${CYCLES_API_KEY}
cycles.tenant=acme
cycles.workspace=production
cycles.app=support-bot
cycles.http.connect-timeout=2s
cycles.http.read-timeout=5s
cycles.retry.enabled=true
cycles.retry.max-attempts=5
cycles.retry.initial-delay=500ms
cycles.retry.multiplier=2.0
cycles.retry.max-delay=30s
cycles.retry.flush-timeout=10s
cycles.journal.enabled=true
# cycles.journal.dir=/var/lib/my-app/cycles-commit-journal
```

## Auto-configured beans

The starter auto-configures the following beans, all with `@ConditionalOnMissingBean` so you can override any of them:

| Bean | Type | Purpose |
|---|---|---|
| `cyclesWebClient` | `WebClient` | HTTP client with configured timeouts |
| `cyclesClient` | `CyclesClient` | Protocol client (`DefaultCyclesClient`) |
| `evaluator` | `CyclesExpressionEvaluator` | SpEL evaluator |
| `cyclesRequestBuilderService` | `CyclesRequestBuilderService` | Builds protocol request bodies |
| `cyclesValueResolutionService` | `CyclesValueResolutionService` | Resolves Subject field values |
| `retryEngine` | `CommitRetryEngine` | Handles durable settlement replay (`JournaledCommitRetryEngine`) |
| `cyclesLifecycleService` | `CyclesLifecycleService` | Orchestrates the full lifecycle |
| `aspect` | `CyclesAspect` | AOP aspect for `@Cycles` annotation |
| `cyclesSelfInvocationDetector` | `CyclesSelfInvocationDetector` | Bean post-processor (declared as a `static` `@Bean`) that warns at startup about beans susceptible to the self-invocation pitfall |

### Overriding a bean

To replace any auto-configured bean, define your own:

```java
@Configuration
public class CustomCyclesConfig {

    @Bean
    public CyclesClient cyclesClient() {
        // Your custom implementation
        return new MyCustomCyclesClient();
    }
}
```

The auto-configuration will skip creating its default `CyclesClient` when it detects yours.

## Environment-specific configuration

### Using Spring profiles

Create profile-specific files in `src/main/resources/`:

```yaml
# application.yml (shared)
cycles:
  tenant: acme
  retry:
    enabled: true

---
# application-dev.yml
cycles:
  base-url: http://localhost:7878
  api-key: dev-key

---
# application-prod.yml
cycles:
  base-url: https://cycles.internal.example.com
  api-key: ${CYCLES_API_KEY}
  http:
    read-timeout: 10s
```

### Using environment variables

Every property can be set via environment variables using Spring Boot's relaxed binding:

| Property | Environment variable |
|---|---|
| `cycles.base-url` | `CYCLES_BASE_URL` |
| `cycles.api-key` | `CYCLES_API_KEY` |
| `cycles.tenant` | `CYCLES_TENANT` |
| `cycles.http.connect-timeout` | `CYCLES_HTTP_CONNECT_TIMEOUT` |
| `cycles.retry.max-attempts` | `CYCLES_RETRY_MAX_ATTEMPTS` |
| `cycles.retry.flush-timeout` | `CYCLES_RETRY_FLUSH_TIMEOUT` |
| `cycles.journal.enabled` | `CYCLES_JOURNAL_ENABLED` |
| `cycles.journal.dir` | `CYCLES_JOURNAL_DIR` |

## Next steps

- [Getting Started with the Spring Boot Starter](/quickstart/getting-started-with-the-cycles-spring-boot-starter) — quick start guide
- [SpEL Expression Reference](/configuration/spel-expression-reference-for-cycles) — expression syntax
- [Custom Field Resolvers](/how-to/custom-field-resolvers-in-cycles) — dynamic Subject field resolution
- [SDK Settlement Recovery and Durability](/protocol/sdk-settlement-recovery-and-durability) — journal, replay, expiry fallback, and guarantee boundary
- [Server Configuration Reference](/configuration/server-configuration-reference-for-cycles) — server-side properties



# Python Client Configuration Reference

This is the complete reference for all configuration options available in the `runcycles` Python client.

## CyclesConfig

All configuration is provided through the `CyclesConfig` dataclass.

### Required fields

| Field | Type | Description |
|---|---|---|
| `base_url` | `str` | Base URL of the Cycles server (e.g., `http://localhost:7878`) |
| `api_key` | `str` | API key for authentication |

### Subject defaults

These fields set default values for the Subject used in `@cycles` decorators. They apply to all decorated functions unless overridden at the decorator level.

| Field | Type | Default | Description |
|---|---|---|---|
| `tenant` | `str \| None` | `None` | Default tenant |
| `workspace` | `str \| None` | `None` | Default workspace |
| `app` | `str \| None` | `None` | Default application name |
| `workflow` | `str \| None` | `None` | Default workflow |
| `agent` | `str \| None` | `None` | Default agent |
| `toolset` | `str \| None` | `None` | Default toolset |

### HTTP timeouts

| Field | Type | Default | Description |
|---|---|---|---|
| `connect_timeout` | `float` | `2.0` | TCP connection timeout in seconds |
| `read_timeout` | `float` | `5.0` | Read timeout for responses in seconds |

### Retry configuration

Controls the commit retry engine and its bounded process-exit drain.

| Field | Type | Default | Description |
|---|---|---|---|
| `retry_enabled` | `bool` | `True` | Enable automatic commit retries |
| `retry_max_attempts` | `int` | `5` | Maximum number of retry attempts |
| `retry_initial_delay` | `float` | `0.5` | Delay before the first retry (seconds) |
| `retry_multiplier` | `float` | `2.0` | Backoff multiplier between retries |
| `retry_max_delay` | `float` | `30.0` | Maximum delay between retries (seconds) |
| `retry_flush_timeout` | `float` | `10.0` | Process-wide wait at interpreter exit for in-flight settlement retries (seconds); `0` disables the wait |

#### How retry works

When settlement fails transiently, the retry engine schedules a same-key retry using exponential backoff:

```
Attempt 1: wait 0.5s
Attempt 2: wait 1.0s
Attempt 3: wait 2.0s
Attempt 4: wait 4.0s
Attempt 5: wait 8.0s (capped at max_delay)
```

HTTP 429 honors a valid `Retry-After` floor, capped by the SDK's bounded-delay policy. Authentication failures and unclassifiable 4xx responses stop the current retry run but retain the durable record. A genuine, understood client rejection stops retrying and removes the record.

### Durable journal

| Field | Type | Default | Description |
|---|---|---|---|
| `journal_enabled` | `bool` | `True` | Persist unresolved known-actual settlement across process restarts |
| `journal_dir` | `str \| None` | `None` | Journal base directory; `None` uses `~/.runcycles/commit-journal` |

The lifecycle helpers persist known actual usage before the first commit request. A schema-valid HTTP `200` commit or schema-valid HTTP `201` event proves success; ambiguous outcomes, retry exhaustion, authentication failures, and unclassifiable 4xx responses remain journaled for replay. An expired commit switches to `POST /v1/events` with the original idempotency key.

The journal is partitioned by server and principal. Configure `tenant` so pending records remain discoverable after API-key rotation. Journal records do not store API keys, but they do contain settlement bodies and metadata; protect the directory as sensitive application state.

See [SDK Settlement Recovery and Durability](/protocol/sdk-settlement-recovery-and-durability) for the guarantee boundary and recovery choreography.

## Programmatic configuration

```python
from runcycles import CyclesConfig

config = CyclesConfig(
    # Required
    base_url="http://localhost:7878",
    api_key="cyc_live_...",

    # Subject defaults
    tenant="acme",
    workspace="production",
    app="support-bot",

    # HTTP settings
    connect_timeout=2.0,
    read_timeout=5.0,

    # Commit retry
    retry_enabled=True,
    retry_max_attempts=5,
    retry_initial_delay=0.5,
    retry_multiplier=2.0,
    retry_max_delay=30.0,
    retry_flush_timeout=10.0,

    # Durable settlement journal
    journal_enabled=True,
    journal_dir=None,  # ~/.runcycles/commit-journal
)
```

## Environment variable configuration

Use `CyclesConfig.from_env()` to load configuration from environment variables. The default prefix is `CYCLES_`:

```python
config = CyclesConfig.from_env()
```

| Environment variable | Maps to | Required |
|---|---|---|
| `CYCLES_BASE_URL` | `base_url` | Yes |
| `CYCLES_API_KEY` | `api_key` | Yes |
| `CYCLES_TENANT` | `tenant` | No |
| `CYCLES_WORKSPACE` | `workspace` | No |
| `CYCLES_APP` | `app` | No |
| `CYCLES_WORKFLOW` | `workflow` | No |
| `CYCLES_AGENT` | `agent` | No |
| `CYCLES_TOOLSET` | `toolset` | No |
| `CYCLES_CONNECT_TIMEOUT` | `connect_timeout` | No |
| `CYCLES_READ_TIMEOUT` | `read_timeout` | No |
| `CYCLES_RETRY_ENABLED` | `retry_enabled` | No |
| `CYCLES_RETRY_MAX_ATTEMPTS` | `retry_max_attempts` | No |
| `CYCLES_RETRY_INITIAL_DELAY` | `retry_initial_delay` | No |
| `CYCLES_RETRY_MULTIPLIER` | `retry_multiplier` | No |
| `CYCLES_RETRY_MAX_DELAY` | `retry_max_delay` | No |
| `CYCLES_RETRY_FLUSH_TIMEOUT` | `retry_flush_timeout` | No |
| `CYCLES_JOURNAL_ENABLED` | `journal_enabled` | No |
| `CYCLES_JOURNAL_DIR` | `journal_dir` | No |

A custom prefix can be passed: `CyclesConfig.from_env(prefix="MY_PREFIX_")`.

## `@cycles` decorator parameters

The `@cycles` decorator accepts parameters that control reservation behavior per-call. These are separate from the `CyclesConfig` connection settings above. For full documentation and examples, see [Getting Started with the Python Client — Decorator parameters](/quickstart/getting-started-with-the-python-client#decorator-parameters).

| Parameter | Type | Default | Description |
|---|---|---|---|
| `estimate` | `int \| Callable` | (required) | Estimated cost. Int constant or callable receiving the function's `*args, **kwargs`. |
| `actual` | `int \| Callable \| None` | `None` | Actual cost. Int constant or callable receiving the return value. Defaults to estimate. |
| `action_kind` | `str \| Callable[..., str \| None] \| None` | `None` | Action category (e.g. `"llm.completion"`). |
| `action_name` | `str \| Callable[..., str \| None] \| None` | `None` | Action identifier (e.g. `"gpt-4"`). |
| `action_tags` | `list[str] \| Callable[..., list[str] \| None] \| None` | `None` | Tags for filtering and reporting. |
| `unit` | `Unit \| str` | `USD_MICROCENTS` | Budget unit: `USD_MICROCENTS`, `TOKENS`, `CREDITS`, `RISK_POINTS`. |
| `ttl_ms` | `int` | `60000` | Reservation TTL in milliseconds (range: 1,000–86,400,000). |
| `grace_period_ms` | `int \| None` | `None` | Grace period after TTL expiry in milliseconds. When `None`, the server applies its default (5000ms). Valid range: 0–60,000. |
| `overage_policy` | `str` | `"ALLOW_IF_AVAILABLE"` | `"REJECT"`, `"ALLOW_IF_AVAILABLE"`, or `"ALLOW_WITH_OVERDRAFT"`. |
| `dry_run` | `bool` | `False` | If `True`, evaluate without persisting. Function does not execute. |
| `tenant` | `str \| Callable[..., str \| None] \| None` | `None` | Subject tenant override (takes precedence over config default). |
| `workspace` | `str \| Callable[..., str \| None] \| None` | `None` | Subject workspace override. |
| `app` | `str \| Callable[..., str \| None] \| None` | `None` | Subject app override. |
| `workflow` | `str \| Callable[..., str \| None] \| None` | `None` | Subject workflow override. |
| `agent` | `str \| Callable[..., str \| None] \| None` | `None` | Subject agent override. |
| `toolset` | `str \| Callable[..., str \| None] \| None` | `None` | Subject toolset override. |
| `dimensions` | `dict[str, str] \| Callable[..., dict[str, str] \| None] \| None` | `None` | Custom dimensions for the subject. |
| `client` | `CyclesClient \| AsyncCyclesClient \| None` | `None` | Explicit client. Falls back to module-level default. |
| `use_estimate_if_actual_not_provided` | `bool` | `True` | If `True` and `actual` is `None`, use estimate as actual at commit. |

### Callable resolution semantics

Subject fields, `action_kind`, `action_name`, `action_tags`, and `dimensions` accept a callable in place of a constant (since 0.4.0). The callable is resolved on every call, invoked with the decorated function's `*args, **kwargs` at reservation time. A `None` (or otherwise falsy) result falls through:

- **Subject fields** (`tenant`, `workspace`, `app`, `workflow`, `agent`, `toolset`) — fall back to the `CyclesConfig` default; omitted if that is also unset.
- **`action_kind` / `action_name`** — fall back to `"unknown"`.
- **`action_tags` / `dimensions`** — omitted from the request.

## Setting a default client

Instead of passing `client=` to every `@cycles` decorator, set a module-level default:

```python
from runcycles import CyclesClient, set_default_client, set_default_config

# Option 1: Set a config (client created lazily)
set_default_config(config)

# Option 2: Set an explicit client
set_default_client(CyclesClient(config))
```

## Resolution order

For each Subject field, the decorator resolves the value using this priority:

1. **Decorator parameter** — if set on the `@cycles` decorator, it wins. If the parameter is a callable, it is invoked with the function's `*args, **kwargs` on each call; a falsy result falls through to the next step.
2. **Config default** — if set on the `CyclesConfig` instance

If neither provides a value, the field is omitted from the request.

## Disabling retry

```python
config = CyclesConfig(
    base_url="http://localhost:7878",
    api_key="cyc_live_...",
    retry_enabled=False,
)
```

This disables active background retries, not durability. Failed known-actual settlement remains journaled for replay on a later run while `journal_enabled=True`. Disable both only when the application supplies equivalent durable settlement recovery.

## Aggressive retry for critical commits

```python
config = CyclesConfig(
    base_url="http://localhost:7878",
    api_key="cyc_live_...",
    retry_max_attempts=10,
    retry_initial_delay=0.2,
    retry_multiplier=1.5,
    retry_max_delay=60.0,
)
```

## Next steps

- [Getting Started with the Python Client](/quickstart/getting-started-with-the-python-client) — quick start guide
- [Error Handling in Python](/how-to/error-handling-patterns-in-python) — exception handling patterns
- [Using the Client Programmatically](/how-to/using-the-cycles-client-programmatically) — direct client usage
- [SDK Settlement Recovery and Durability](/protocol/sdk-settlement-recovery-and-durability) — journal, replay, expiry fallback, and guarantee boundary
- [Server Configuration Reference](/configuration/server-configuration-reference-for-cycles) — server-side properties



# Rust Client Configuration Reference

Complete reference for all configuration options in the `runcycles` Rust client. Targets `runcycles >= 0.3.2`. The async client is the default; the blocking variant is available behind a feature flag.

For the introductory walkthrough, see the [Rust Client Quickstart](/quickstart/getting-started-with-the-rust-client). For runtime error patterns, see [Error Handling in Rust](/how-to/error-handling-patterns-in-rust).

## CyclesConfig

The `CyclesConfig` struct holds all client configuration. It can be constructed via the builder API (recommended), via `CyclesConfig::from_env()`, or by populating the struct fields directly.

### Required fields

| Field | Type | Description |
|---|---|---|
| `base_url` | `String` | Base URL of the [Cycles server](/glossary#cycles-server) (e.g. `http://localhost:7878`) |
| `api_key` | `String` | API key for authentication. [Tenant](/glossary#tenant)-scoped key starting with `cyc_live_` |

### Subject defaults

These fields hold subject values that are stored on the config and available via `client.config()`. **They are not auto-applied to per-request subjects in 0.3.x** — see [Subject defaults: what they do (and don't)](#subject-defaults-what-they-do-and-don-t) below for the actual behavior.

| Field | Type | Default | Description |
|---|---|---|---|
| `tenant` | `Option<String>` | `None` | Default tenant |
| `workspace` | `Option<String>` | `None` | Default workspace |
| `app` | `Option<String>` | `None` | Default application name |
| `workflow` | `Option<String>` | `None` | Default workflow |
| `agent` | `Option<String>` | `None` | Default agent |
| `toolset` | `Option<String>` | `None` | Default toolset |

### HTTP timeouts

| Field | Type | Default | Description |
|---|---|---|---|
| `connect_timeout` | `Duration` | `2_000 ms` | TCP connection timeout |
| `read_timeout` | `Duration` | `5_000 ms` | Read timeout for responses |

`Duration` values are constructed with `std::time::Duration::from_millis(...)` or `from_secs(...)` in programmatic configuration. Environment variables are expressed in milliseconds (see below).

### Retry configuration

`ReservationGuard::commit()` retries transient failures inline and reuses the original `CommitRequest` and idempotency key. HTTP 429 honors `Retry-After`. The heartbeat remains active while inline retries run.

| Field | Type | Default | Description |
|---|---|---|---|
| `retry_enabled` | `bool` | `true` | Enable automatic commit retries |
| `retry_max_attempts` | `u32` | `5` | Maximum retry attempts after the initial commit attempt |
| `retry_initial_delay` | `Duration` | `500 ms` | Delay before the first retry |
| `retry_multiplier` | `f64` | `2.0` | Exponential backoff multiplier between retries |
| `retry_max_delay` | `Duration` | `30_000 ms` | Maximum delay between retries |

Known actual usage is durably journaled before the first commit request. If the retry schedule is exhausted, authentication fails, or a client response remains ambiguous, `commit()` returns `Error::CommitPending` and leaves the record queued for same-key replay. Do not compensate with a different key.

### Durable journal

| Field | Type | Default | Description |
|---|---|---|---|
| `journal_enabled` | `bool` | `true` | Persist unresolved known-actual settlement across process restarts |
| `journal_dir` | `Option<PathBuf>` | `None` | Journal base directory; `None` uses `~/.runcycles/commit-journal` |

Only a schema-valid HTTP `200` commit or schema-valid HTTP `201` event proves success. If commit returns HTTP 410 or `RESERVATION_EXPIRED`, the guard switches the journal to event mode before calling `POST /v1/events` with the original key. Unresolved records replay automatically when a client is created inside a Tokio runtime.

Use `flush_pending_commits_with_timeout(...)` for a bounded startup or graceful-shutdown drain. The blocking client exposes the same operation. Timed-out or failed records remain on disk.

The journal is partitioned by server and principal. Configure `tenant` so pending records remain discoverable after API-key rotation. API keys are not stored in records, but settlement bodies and metadata are; protect the directory as sensitive application state.

## Programmatic configuration

The builder API is the recommended way to construct a client. It exposes connection settings, subject defaults, and every commit-retry setting.

```rust
use runcycles::CyclesClient;
use std::time::Duration;

let client = CyclesClient::builder(
    "cyc_live_...",
    "http://localhost:7878",
)
.tenant("acme-corp")
.workspace("production")
.app("support-bot")
.connect_timeout(Duration::from_millis(2_000))
.read_timeout(Duration::from_millis(5_000))
.retry_enabled(true)
.retry_max_attempts(5)
.retry_initial_delay(Duration::from_millis(500))
.retry_multiplier(2.0)
.retry_max_delay(Duration::from_secs(30))
.journal_enabled(true)
.journal_dir("/var/lib/my-app/cycles-commit-journal")
.build();
```

You can also construct `CyclesConfig` directly:

```rust
use runcycles::{CyclesClient, CyclesConfig};
use std::time::Duration;

let config = CyclesConfig {
    base_url: "http://localhost:7878".into(),
    api_key: "cyc_live_...".into(),
    tenant: Some("acme-corp".into()),
    workspace: None,
    app: None,
    workflow: None,
    agent: None,
    toolset: None,
    connect_timeout: Duration::from_millis(2_000),
    read_timeout: Duration::from_millis(5_000),
    retry_enabled: true,
    retry_max_attempts: 5,
    retry_initial_delay: Duration::from_millis(500),
    retry_multiplier: 2.0,
    retry_max_delay: Duration::from_secs(30),
    journal_enabled: true,
    journal_dir: None,
};

let client = CyclesClient::new(config);
```

`CyclesConfig` does not implement `Default`; populate every field explicitly when constructing the struct directly, or use the builder.

## Environment variable configuration

Use `CyclesConfig::from_env()` to load configuration from environment variables. The default prefix is `CYCLES_`:

```rust
use runcycles::CyclesConfig;

let config = CyclesConfig::from_env().expect("missing required CYCLES_* env vars");
```

| Environment variable | Maps to | Type | Required |
|---|---|---|---|
| `CYCLES_BASE_URL` | `base_url` | string | Yes |
| `CYCLES_API_KEY` | `api_key` | string | Yes |
| `CYCLES_TENANT` | `tenant` | string | No |
| `CYCLES_WORKSPACE` | `workspace` | string | No |
| `CYCLES_APP` | `app` | string | No |
| `CYCLES_WORKFLOW` | `workflow` | string | No |
| `CYCLES_AGENT` | `agent` | string | No |
| `CYCLES_TOOLSET` | `toolset` | string | No |
| `CYCLES_CONNECT_TIMEOUT` | `connect_timeout` | milliseconds (integer) | No |
| `CYCLES_READ_TIMEOUT` | `read_timeout` | milliseconds (integer) | No |
| `CYCLES_RETRY_ENABLED` | `retry_enabled` | `true` / `false` | No |
| `CYCLES_RETRY_MAX_ATTEMPTS` | `retry_max_attempts` | integer | No |
| `CYCLES_RETRY_INITIAL_DELAY` | `retry_initial_delay` | milliseconds (integer) | No |
| `CYCLES_RETRY_MULTIPLIER` | `retry_multiplier` | float | No |
| `CYCLES_RETRY_MAX_DELAY` | `retry_max_delay` | milliseconds (integer) | No |
| `CYCLES_JOURNAL_ENABLED` | `journal_enabled` | `true` / `false` | No |
| `CYCLES_JOURNAL_DIR` | `journal_dir` | filesystem path | No |

::: tip Custom env var prefix
The Rust client supports loading from a custom prefix, which is useful when a single process holds connections to multiple Cycles instances:

```rust
let primary  = CyclesConfig::from_env_with_prefix("CYCLES_PRIMARY_")?;
let staging  = CyclesConfig::from_env_with_prefix("CYCLES_STAGING_")?;
```

The default `from_env()` is equivalent to `from_env_with_prefix("CYCLES_")`.
:::

## Subject defaults: what they do (and don't)

The subject fields on `CyclesConfig` (`tenant`, `workspace`, `app`, `workflow`, `agent`, `toolset`) are stored on the config and accessible via `client.config()`, but the high-level helpers in `runcycles` 0.3.x **do not automatically apply them** to the per-request `Subject`. Each `with_cycles()` / `client.reserve()` / `client.create_reservation()` call uses the `Subject` you pass in explicitly (or `Subject::default()` if you pass none).

If you want a single tenant applied to every request, build the subject once and reuse it:

```rust
use runcycles::models::Subject;

let subject = Subject {
    tenant: Some("acme-corp".into()),
    workspace: Some("production".into()),
    ..Default::default()
};

// Pass the same subject to every WithCyclesConfig / ReservationCreateRequest
let cfg = WithCyclesConfig::new(Amount::tokens(1_000))
    .action("llm.completion", "gpt-4o-mini")
    .subject(subject.clone());
```

Future versions of the crate may wire the config's subject defaults into request subjects automatically; this reference will be updated when that lands.

## Custom `reqwest::Client`

By default, the client creates its own `reqwest::Client` with the configured timeouts. Pass a custom one when you need shared connection pooling, custom middleware, TLS pinning, or proxy support:

```rust
use runcycles::CyclesClient;
use reqwest::Client;
use std::time::Duration;

let http = Client::builder()
    .pool_max_idle_per_host(20)
    .timeout(Duration::from_secs(10))
    .build()?;

let client = CyclesClient::builder(
    "cyc_live_...",
    "http://localhost:7878",
)
.http_client(http)  // overrides connect_timeout / read_timeout from config
.tenant("acme-corp")
.build();
```

When a custom `reqwest::Client` is provided, the config's `connect_timeout` and `read_timeout` are ignored — set them on the `reqwest::Client` instead.

## Blocking client variant

For applications running in synchronous contexts (CLI tools, sync HTTP frameworks like `rouille`, embedded scripts), the crate ships a blocking variant behind a feature flag.

```toml
# Cargo.toml
[dependencies]
runcycles = { version = "0.3", features = ["blocking"] }
```

```rust
use runcycles::{BlockingCyclesClient, CyclesConfig, models::BalanceParams};

let client = BlockingCyclesClient::new(CyclesConfig::from_env()?)?;
let resp = client.get_balances(&BalanceParams {
    tenant: Some("acme-corp".into()),
    ..Default::default()
})?;
```

The blocking client exposes the low-level protocol methods — `create_reservation`, `create_reservation_with_metadata`, `commit_reservation`, `release_reservation`, `extend_reservation`, `decide`, `create_event`, `list_reservations`, `get_reservation`, `get_balances` — plus `config()` and pending-journal flush operations. The high-level `with_cycles()` helper and `ReservationGuard` RAII pattern are async-only in 0.3.x; blocking callers compose the reserve / commit / release sequence and must persist their application-owned settlement requests themselves.

::: warning Don't mix runtimes
The blocking client must not be called from inside a Tokio runtime (it will block the executor). For most applications using `tokio::main`, the async client is correct. The blocking variant is for genuinely synchronous contexts.
:::

## CyclesClientBuilder method reference

| Method | Sets | Notes |
|---|---|---|
| `new(api_key, base_url)` | required fields | The constructor; both args are `impl Into<String>` |
| `.tenant(s)` | config subject default | All subject methods accept `impl Into<String>`. Stored on the config but not auto-applied to request subjects — see [Subject defaults](#subject-defaults-what-they-do-and-don-t). |
| `.workspace(s)` | config subject default | |
| `.app(s)` | config subject default | |
| `.workflow(s)` | config subject default | |
| `.agent(s)` | config subject default | |
| `.toolset(s)` | config subject default | |
| `.connect_timeout(d)` | HTTP | Takes `std::time::Duration` |
| `.read_timeout(d)` | HTTP | Takes `std::time::Duration` |
| `.retry_enabled(b)` | commit retry | Enables or disables inline commit retry |
| `.retry_max_attempts(n)` | commit retry | Sets retry attempts after the initial commit attempt |
| `.retry_initial_delay(d)` | commit retry | Sets the delay before the first retry |
| `.retry_multiplier(n)` | commit retry | Sets the exponential backoff multiplier |
| `.retry_max_delay(d)` | commit retry | Caps the delay between retries |
| `.journal_enabled(b)` | durability | Enables or disables the pending-settlement journal |
| `.journal_dir(path)` | durability | Sets the journal base directory |
| `.http_client(c)` | HTTP | Provide a custom `reqwest::Client`; overrides timeouts |
| `.build()` | finalizes | Returns `CyclesClient` (async) |
| `.build_blocking()` | finalizes | Returns `Result<BlockingCyclesClient, Error>`; requires the `blocking` feature |

## Next steps

- [Rust Client Quickstart](/quickstart/getting-started-with-the-rust-client) — installation and first [reservation](/glossary#reservation)
- [Error Handling in Rust](/how-to/error-handling-patterns-in-rust) — retry, recovery, and [graceful degradation](/glossary#graceful-degradation)
- [Integrating Cycles with Rust](/how-to/integrating-cycles-with-rust) — multi-step flows, streaming, framework integration
- [SDK Settlement Recovery and Durability](/protocol/sdk-settlement-recovery-and-durability) — journal, replay, expiry fallback, and guarantee boundary
- [Server Configuration Reference](/configuration/server-configuration-reference-for-cycles) — server-side properties
- [How Reserve-Commit Works](/protocol/how-reserve-commit-works-in-cycles) — the underlying lifecycle



# Server Configuration Reference for Cycles

This reference covers the Cycles-defined properties and deployment-facing Spring settings in the current runtime, admin, and events service implementations. Standard Spring Boot properties that the services do not set explicitly are outside its scope.

The server uses Spring Boot's configuration system. Properties can be set in `application.properties`, `application.yml`, or via environment variables.

## Server properties

| Property | Default | Env Variable | Description |
|---|---|---|---|
| `server.port` | `7878` | `SERVER_PORT` | HTTP port the server listens on |
| `spring.application.name` | `cycles-protocol-service` | — | Application name |
| `spring.task.scheduling.pool.size` | `4` | `CYCLES_SCHEDULER_POOL_SIZE` | Bounded scheduler used by expiry, audit, event, and reservation-index maintenance jobs |
| `server.compression.enabled` | `true` | — | Enable HTTP response compression |
| `server.compression.min-response-size` | `1024` | — | Minimum response size in bytes before compression |
| `server.shutdown` | `graceful` | — | Enable graceful shutdown |
| `spring.lifecycle.timeout-per-shutdown-phase` | `30s` | — | Maximum graceful-shutdown phase duration |

## Redis connection

| Property | Default | Env Variable | Description |
|---|---|---|---|
| `redis.host` | `localhost` | `REDIS_HOST` | Redis server hostname |
| `redis.port` | `6379` | `REDIS_PORT` | Redis server port |
| `redis.password` | (empty) | `REDIS_PASSWORD` | Redis password (optional) |
| `redis.pool.max-total` | `128` | — | JedisPool max active connections |
| `redis.pool.max-idle` | `32` | — | JedisPool max idle connections |
| `redis.pool.min-idle` | `16` | — | JedisPool min idle connections kept warm |
| `redis.pool.max-wait-ms` | `2000` | — | Max ms a caller waits for a pooled connection before `JedisException` |

Redis 7+ is required for Lua script compatibility. Tune `redis.pool.max-total` upward on high-concurrency instances — the reservation Lua script holds a connection for the duration of the atomic script call.

## Reservation expiry

| Property | Default | Env Variable | Description |
|---|---|---|---|
| `cycles.expiry.interval-ms` | `5000` | `CYCLES_EXPIRY_INTERVAL_MS` | How often the background expiry sweep runs (ms) |

The expiry sweep scans for reservations past their TTL and marks them as `EXPIRED`, releasing their reserved budget back to the affected scopes.

### Tuning the sweep interval

- **Lower values** (e.g., 1000ms): expired reservations are cleaned up faster, budget is returned sooner. Increases Redis load slightly.
- **Higher values** (e.g., 30000ms): less Redis overhead, but expired reservations hold budget longer before cleanup.

For most deployments, the default 5000ms is a good balance.

## Distributed maintenance and reservation index

Runtime maintenance jobs coordinate across replicas with renewable, owner-safe Redis leases. The optional per-tenant created-at index is dual-written by all writers but remains disabled for reads and repair by default so it can be rolled out safely.

| Property | Default | Env Variable | Description |
|---|---|---|---|
| `cycles.maintenance.lease-ttl-ms` | `30000` | `CYCLES_MAINTENANCE_LEASE_TTL_MS` | Lease TTL for each distributed maintenance job |
| `cycles.maintenance.renew-interval-ms` | `10000` | `CYCLES_MAINTENANCE_RENEW_INTERVAL_MS` | Lease-renewal interval; must remain below the lease TTL |
| `cycles.reservation-index.created-at.enabled` | `false` | `RESERVATION_CREATED_AT_INDEX_ENABLED` | Enable reads and repair for the per-tenant created-at reservation index |
| `cycles.reservation-index.created-at.repair-interval-ms` | `300000` | `RESERVATION_CREATED_AT_INDEX_REPAIR_INTERVAL_MS` | Delay between repair batches |
| `cycles.reservation-index.created-at.initial-delay-ms` | `5000` | `RESERVATION_CREATED_AT_INDEX_INITIAL_DELAY_MS` | Initial delay before repair begins |
| `cycles.reservation-index.created-at.failure-backoff-ms` | `3600000` | `RESERVATION_CREATED_AT_INDEX_FAILURE_BACKOFF_MS` | Backoff after a repair failure |
| `cycles.reservation-index.created-at.sweep-cron` | `0 45 3 * * *` | `RESERVATION_CREATED_AT_INDEX_SWEEP_CRON` | Cron for stale index-pointer cleanup |

## Runtime cross-plane settings

| Property | Default | Env Variable | Description |
|---|---|---|---|
| `admin.api-key` | (empty) | `ADMIN_API_KEY` | Admin key accepted on the allowlisted admin-on-behalf-of and protected operational endpoints |
| `webhook.secret.encryption-key` | (empty) | `WEBHOOK_SECRET_ENCRYPTION_KEY` | Shared base64 AES-256 key. Production Compose requires it; use the same value on admin and events. |
| `events.retention.event-ttl-days` | `90` | `EVENT_TTL_DAYS` | Runtime-emitted event record TTL |
| `events.retention.delivery-ttl-days` | `14` | `DELIVERY_TTL_DAYS` | Delivery record TTL stamped for the shared event plane |
| `events.retention.sweep-cron` | `0 30 3 * * *` | `EVENT_RETENTION_SWEEP_CRON` | Cron for stale event/delivery index cleanup |
| `cycles.evidence.queue.pending-key` | `evidence:pending` | `EVIDENCE_PENDING_KEY` | Source queue consumed by the events-service evidence worker |
| `cycles.evidence.store.key-prefix` | `evidence:envelope:` | `EVIDENCE_STORE_KEY_PREFIX` | Redis key prefix used by public evidence retrieval; must match the events service |

## Public endpoint rate limiting (v0.1.25.46)

The runtime server applies a fixed-window per-client-IP rate limit to the **public (unauthenticated)** endpoints only — `GET /v1/evidence/*` and the CyclesEvidence JWKS — implementing the spec's SHOULD-level 429 throttling (`error=LIMIT_EXCEEDED` with `Retry-After` and `X-RateLimit-Reset`). Authenticated `/v1` endpoints are not covered.

| Property | Default | Env Variable | Description |
|---|---|---|---|
| `cycles.public-rate-limit.enabled` | `true` | `CYCLES_PUBLIC_RATE_LIMIT_ENABLED` | Enable the public-endpoint rate limiter. |
| `cycles.public-rate-limit.requests-per-minute` | `300` | `CYCLES_PUBLIC_RATE_LIMIT_REQUESTS_PER_MINUTE` | Fixed 60s window per client IP, per instance. Keyed on the socket peer address — behind an ingress that terminates connections, prefer rate limiting there and/or raise this limit. |

## Event emission (v0.1.25.45)

The runtime emits webhook/event side effects through a bounded, non-blocking executor so a dispatch Redis outage or slow event persistence cannot grow heap without limit.

| Property | Default | Env Variable | Description |
|---|---|---|---|
| `cycles.events.emit.threads` | `0` | `CYCLES_EVENTS_EMIT_THREADS` | Worker threads for the non-blocking runtime event emitter. |
| `cycles.events.emit.queue-capacity` | `10000` | `CYCLES_EVENTS_EMIT_QUEUE_CAPACITY` | Bounded queue capacity; under sustained event-persistence outage, side effects past this bound are dropped (ledger mutations are unaffected). |

## Runtime audit log retention (v0.1.25.15)

The runtime server writes audit entries for admin-on-behalf-of operations (force-release) to `audit:log:{id}` keys in Redis. v0.1.25.15 adds TTL-based retention so these rows respect the same 400-day authenticated tier as the admin plane.

| Property | Default | Env Variable | Description |
|---|---|---|---|
| `audit.retention.days` | `400` | `AUDIT_RETENTION_DAYS` | TTL in days for runtime-written audit rows. Set to `0` for indefinite retention (legal hold, HIPAA-adjacent deployments, environments that offload to an archive store). |
| `audit.sweep.cron` | `0 0 3 * * *` | `AUDIT_SWEEP_CRON` | Cron for the daily `@Scheduled` sweep that prunes expired `audit:logs:{tenantId}` and `audit:logs:_all` ZSET pointers. Safe to run alongside admin's sweep (idempotent `ZREMRANGEBYSCORE`). |

Runtime audit rows never use the admin-plane `__admin__` / `__unauth__` sentinels — the runtime never fails pre-auth for admin keys, so a single tier is sufficient.

## Metrics

| Property | Default | Env Variable | Description |
|---|---|---|---|
| `cycles.metrics.tenant-tag.enabled` | `true` | `CYCLES_METRICS_TENANT_TAG_ENABLED` | When `true`, Prometheus counters include a `tenant` label. Set to `false` in deployments with many thousands of tenants to bound series cardinality. |

The runtime server publishes 11 domain counters plus one maintenance timer; the events service publishes 17 delivery, evidence, dispatcher, and security counters plus one delivery-latency timer. The `tenant-tag.enabled` toggle is mirrored on both services, but the defaults differ: `true` on the runtime and `false` on the events service. For the complete current inventory, tag definitions, scrape targets, and alert recipes, see [Prometheus Metrics Reference](/how-to/prometheus-metrics-reference).

## JSON serialization

| Property | Default | Description |
|---|---|---|
| `spring.jackson.serialization.write-dates-as-timestamps` | `false` | Dates are ISO-8601 strings, not timestamps |
| `spring.jackson.deserialization.fail-on-unknown-properties` | `true` | Reject requests with unknown fields |
| `spring.jackson.default-property-inclusion` | `non_null` | Omit null fields from responses |

These settings enforce strict request validation and clean responses.

## Logging

| Property | Default | Description |
|---|---|---|
| `logging.level.root` | `INFO` | Root log level |
| `logging.level.io.runcycles.protocol` | `INFO` | Cycles-specific log level |
| `logging.pattern.console` | `%d{...} [%thread] %-5level %logger{36} - %msg%n` | Log format |

### Recommended production settings

```properties
logging.level.root=WARN
logging.level.io.runcycles.protocol=INFO
```

### Debugging

For troubleshooting, enable DEBUG on the data layer:

```properties
logging.level.io.runcycles.protocol.data=DEBUG
```

This logs Lua script execution details, scope derivation, and balance calculations.

### Structured (JSON) logging

Cycles does not register a custom JSON log format. Because the services run on Spring Boot 3.4+, you can opt into Spring's built-in structured logging by setting one of the following at deploy time:

| Variable | Value | Description |
|---|---|---|
| `LOGGING_STRUCTURED_FORMAT_CONSOLE` | `ecs` | Emit logs in Elastic Common Schema JSON (Spring Boot built-in). |
| `LOGGING_STRUCTURED_FORMAT_CONSOLE` | `logstash` | Emit logs in Logstash JSON format (Spring Boot built-in). |

When either value is set, Spring Boot overrides `logging.pattern.console` in favor of JSON output. This is stock Spring Boot behavior, not a Cycles-specific feature — the same env var works on the admin and events services.

## OpenAPI / Swagger

| Property | Default | Description |
|---|---|---|
| `springdoc.api-docs.path` | `/api-docs` | Path for the OpenAPI JSON spec |
| `springdoc.swagger-ui.path` | `/swagger-ui.html` | Path for the Swagger UI |
| `springdoc.swagger-ui.enabled` | `true` | Enable Swagger UI |

To disable Swagger UI in production:

```properties
springdoc.swagger-ui.enabled=false
```

The OpenAPI spec at `/api-docs` can remain enabled for tooling.

## Actuator / health checks

| Property | Default | Description |
|---|---|---|
| `management.endpoints.web.exposure.include` | `health,info,prometheus` | Exposed actuator endpoints (runtime server default) |
| `management.endpoint.health.show-details` | `when-authorized` | Show health details |

### Available endpoints (default)

```
GET /actuator/health      — aggregate health check
GET /actuator/info        — application info
GET /actuator/prometheus  — Micrometer metrics in Prometheus exposition format
```

Since v0.1.25.45 (2026-06-27), the runtime server's `OperationalEndpointAuthFilter` protects the operational endpoints with the configured admin key: `/actuator/prometheus`, `/actuator/info`, and aggregate `/actuator/health` require `X-Admin-API-Key`. Only the liveness/readiness probe paths (`/actuator/health/liveness`, `/actuator/health/readiness`) remain unauthenticated for orchestrators. Prometheus scrapers must send `X-Admin-API-Key` on the scrape request, or have a trusted ingress inject it.

### Adding more endpoints

To expose additional actuator endpoints (e.g., `metrics`, `loggers`, `env`):

```properties
management.endpoints.web.exposure.include=health,info,prometheus,metrics,loggers
```

## Security configuration

The server's security is configured in `SecurityConfig.java` plus, since v0.1.25.45 (2026-06-27), `OperationalEndpointAuthFilter.java`, which moved the operational/docs endpoints behind the admin key.

Truly public paths (no key of any kind required):

- `/actuator/health/liveness`, `/actuator/health/readiness` — Kubernetes-style probes
- `/.well-known/**` — Well-known endpoints, including the CyclesEvidence JWKS
- `/v1/evidence/**` — Public evidence retrieval (rate-limited; see [Public endpoint rate limiting](#public-endpoint-rate-limiting-v0-1-25-46))
- `/favicon.ico` — Favicon

Admin-key-protected paths (require `X-Admin-API-Key`):

- `/actuator/**` — All other actuator endpoints, including `/actuator/prometheus`, `/actuator/info`, and aggregate `/actuator/health`
- `/api-docs/**`, `/v3/api-docs/**` — OpenAPI spec
- `/swagger*` — Swagger UI and resources
- `/webjars/**` — WebJar resources

All other paths require a valid `X-Cycles-API-Key` header.

## Representative runtime configuration

This example shows the common deployment settings. Use the tables above for maintenance, evidence, retention, and rate-limit tuning.

```properties
# Server
server.port=7878

# Redis
redis.host=${REDIS_HOST:localhost}
redis.port=${REDIS_PORT:6379}
redis.password=${REDIS_PASSWORD:}

# Expiry
cycles.expiry.interval-ms=5000

# JSON
spring.jackson.serialization.write-dates-as-timestamps=false
spring.jackson.deserialization.fail-on-unknown-properties=true
spring.jackson.default-property-inclusion=non_null

# Logging
logging.level.root=INFO
logging.level.io.runcycles.protocol=INFO

# Swagger
springdoc.api-docs.path=/api-docs
springdoc.swagger-ui.path=/swagger-ui.html
springdoc.swagger-ui.enabled=true

# Actuator
management.endpoints.web.exposure.include=health,info,prometheus
management.endpoint.health.show-details=when-authorized
```

## Environment variable reference

Quick reference for setting all properties via environment variables:

| Variable | Maps to |
|---|---|
| `REDIS_HOST` | `redis.host` |
| `REDIS_PORT` | `redis.port` |
| `REDIS_PASSWORD` | `redis.password` |
| `SERVER_PORT` | `server.port` |
| `CYCLES_EXPIRY_INTERVAL_MS` | `cycles.expiry.interval-ms` |
| `CYCLES_SCHEDULER_POOL_SIZE` | `spring.task.scheduling.pool.size` |
| `CYCLES_MAINTENANCE_LEASE_TTL_MS` | `cycles.maintenance.lease-ttl-ms` |
| `CYCLES_MAINTENANCE_RENEW_INTERVAL_MS` | `cycles.maintenance.renew-interval-ms` |
| `RESERVATION_CREATED_AT_INDEX_ENABLED` | `cycles.reservation-index.created-at.enabled` |
| `RESERVATION_CREATED_AT_INDEX_REPAIR_INTERVAL_MS` | `cycles.reservation-index.created-at.repair-interval-ms` |
| `RESERVATION_CREATED_AT_INDEX_INITIAL_DELAY_MS` | `cycles.reservation-index.created-at.initial-delay-ms` |
| `RESERVATION_CREATED_AT_INDEX_FAILURE_BACKOFF_MS` | `cycles.reservation-index.created-at.failure-backoff-ms` |
| `RESERVATION_CREATED_AT_INDEX_SWEEP_CRON` | `cycles.reservation-index.created-at.sweep-cron` |
| `ADMIN_API_KEY` | `admin.api-key` |
| `WEBHOOK_SECRET_ENCRYPTION_KEY` | `webhook.secret.encryption-key` |
| `EVENT_TTL_DAYS` | `events.retention.event-ttl-days` |
| `DELIVERY_TTL_DAYS` | `events.retention.delivery-ttl-days` |
| `EVENT_RETENTION_SWEEP_CRON` | `events.retention.sweep-cron` |
| `CYCLES_PUBLIC_RATE_LIMIT_ENABLED` | `cycles.public-rate-limit.enabled` |
| `CYCLES_PUBLIC_RATE_LIMIT_REQUESTS_PER_MINUTE` | `cycles.public-rate-limit.requests-per-minute` |
| `CYCLES_EVENTS_EMIT_THREADS` | `cycles.events.emit.threads` |
| `CYCLES_EVENTS_EMIT_QUEUE_CAPACITY` | `cycles.events.emit.queue-capacity` |
| `AUDIT_RETENTION_DAYS` | `audit.retention.days` |
| `AUDIT_SWEEP_CRON` | `audit.sweep.cron` |
| `EVIDENCE_PENDING_KEY` | `cycles.evidence.queue.pending-key` |
| `EVIDENCE_STORE_KEY_PREFIX` | `cycles.evidence.store.key-prefix` |
| `EVIDENCE_SERVER_ID` | `cycles.evidence.server-id` |
| `EVIDENCE_SIGNING_SIGNER_DID` | `cycles.evidence.signing.signer-did` |
| `EVIDENCE_SIGNING_KID` | `cycles.evidence.signing.kid` |
| `EVIDENCE_SIGNING_NBF_MS` | `cycles.evidence.signing.nbf-ms` |
| `EVIDENCE_SIGNING_RETIRED_KEYS` | `cycles.evidence.signing.retired-keys` |

These runtime-server variables configure CyclesEvidence signer-key publication and rotation. They are public identity/JWKS settings; the private `EVIDENCE_SIGNING_PRIVATE_KEY_HEX` lives only on `cycles-server-events` and is not read by `cycles-server`. `KID` / `NBF_MS` describe the active key's JWK; `RETIRED_KEYS` is the JSON rotation history. See [Signer-key resolution and rotation](/protocol/cycles-evidence-envelopes-in-cycles#signer-key-resolution-and-rotation) for the JWK shape and the rotation procedure, and the [identity enablement runbook](https://github.com/runcycles/cycles-server-events/blob/main/docs/evidence-identity-enablement.md) for first-time setup.

---

## Admin Server Configuration

The Cycles Admin Server (`cycles-admin-service`) is a separate service that manages tenants, API keys, budgets, and policies. It runs on port 7979 by default and shares the same Redis instance as the Cycles Server.

### Admin server properties

| Property | Default | Env Variable | Description |
|---|---|---|---|
| `server.port` | `7979` | `SERVER_PORT` | HTTP port the admin server listens on |
| `admin.api-key` | (empty) | `ADMIN_API_KEY` | Master admin key for `X-Admin-API-Key` header |
| `redis.host` | (required) | `REDIS_HOST` | Redis server hostname |
| `redis.port` | (required) | `REDIS_PORT` | Redis server port |
| `redis.password` | (required) | `REDIS_PASSWORD` | Redis password (set empty string if none) |
| `dashboard.cors.origin` | `http://localhost:5173` | `DASHBOARD_CORS_ORIGIN` | Allowed CORS origin for the [admin dashboard](/quickstart/deploying-the-cycles-dashboard). Only needed when the browser calls the admin server directly (dev mode); unused in standard production (nginx reverse-proxies same-origin). |
| `springdoc.swagger-ui.enabled` | `false` | `SWAGGER_ENABLED` | Swagger UI is disabled by default on the admin server; set to `true` to enable. |
| `springdoc.api-docs.enabled` | `false` | `API_DOCS_ENABLED` | OpenAPI JSON spec endpoint (`/api-docs`) is disabled by default on the admin server; set to `true` to enable. |
| `auth.failure-rate-limit.enabled` | `false` | `AUTH_FAILURE_RATE_LIMIT_ENABLED` | Optional in-process guard for repeated 401/403 failures from the same source. Disabled by default for local/test parity; enable in production. |
| `auth.failure-rate-limit.max-per-minute` | `300` | `AUTH_FAILURE_RATE_LIMIT_MAX_PER_MINUTE` | Max auth failures per source per minute before throttling, when the guard is enabled. |
| `auth.failure-rate-limit.max-tracked-sources` | `10000` | `AUTH_FAILURE_RATE_LIMIT_MAX_TRACKED_SOURCES` | Bound the limiter's in-memory source/path buckets; the oldest live bucket is evicted at the cap. |
| `spring.task.scheduling.pool.size` | `2` | `TASK_SCHEDULER_POOL_SIZE` | Scheduler threads. Values below 2 are raised to the enforced safety floor. |
| `tenant-close.reconciler.enabled` | `true` | `TENANT_CLOSE_RECONCILER_ENABLED` | Retry incomplete Mode-B cascades for tenants already marked `CLOSED`. |
| `tenant-close.reconciler.interval-ms` | `300000` | `TENANT_CLOSE_RECONCILER_INTERVAL_MS` | Delay between reconciliation runs. |
| `tenant-close.reconciler.max-tenants-per-run` | `100` | `TENANT_CLOSE_RECONCILER_MAX_TENANTS_PER_RUN` | Maximum due tenant-close work items processed per run. |
| `webhook.secret.encryption-key` | (empty; startup fails) | `WEBHOOK_SECRET_ENCRYPTION_KEY` | Base64 AES-256 key used to encrypt webhook signing secrets. Must match runtime and events. |
| `webhook.secret.allow-plaintext` | `false` | `WEBHOOK_SECRET_ALLOW_PLAINTEXT` | Explicit local/development compatibility escape hatch. With an empty key, `true` permits plaintext and emits a prominent warning. Never enable in production. |
| `events.retention.event-ttl-days` | `90` | `EVENT_TTL_DAYS` | Shared event record TTL. |
| `events.retention.delivery-ttl-days` | `14` | `DELIVERY_TTL_DAYS` | Shared webhook-delivery record TTL. |
| `logging.level.io.runcycles.admin` | `INFO` | `LOG_LEVEL` | Admin-specific log level. |

### Audit log retention

Introduced in `cycles-server-admin` v0.1.25.20 for SOC2-compliant defaults. Failed requests (401/403/400/404/409/500) are now recorded alongside successes; retention is tiered so pre-auth failures expire faster than authenticated entries.

| Property | Default | Env Variable | Description |
|---|---|---|---|
| `audit.retention.authenticated.days` | `400` | `AUDIT_RETENTION_AUTHENTICATED_DAYS` | TTL on authenticated audit entries (success + authenticated failures). `400` covers the SOC2 Type II 12-month lookback + 1-month auditor-engagement buffer. Set to `0` for indefinite retention (legal hold, HIPAA-adjacent). |
| `audit.retention.unauthenticated.days` | `30` | `AUDIT_RETENTION_UNAUTHENTICATED_DAYS` | TTL on pre-auth failures (sentinel tenant `__unauth__`). Enough for brute-force / credential-stuffing post-mortem. Aggregate volume stays visible via Prometheus regardless of TTL. Set to `0` for indefinite. |
| `audit.sample.unauthenticated` | `1` | `AUDIT_SAMPLE_UNAUTHENTICATED` | Sampling rate on unauthenticated entries (`1` = every entry, `100` = 1 in 100). Opt-in hardening against failed-auth floods on internet-exposed admin endpoints. Authenticated entries are **never** sampled. |
| `audit.sweep.cron` | `0 0 3 * * *` | `AUDIT_SWEEP_CRON` | Cron for the daily audit-index sweep. Purges TTL-expired pointers from the `audit:logs:_all` + per-tenant sorted-set indexes. Skipped entirely when `audit.retention.authenticated.days=0`. |

**Alerting.** The Prometheus counter `cycles_admin_audit_writes_total{path_class, outcome}` tracks audit-write health. Alert on `outcome=error` nonzero — audit writes are non-fatal to the request, but silent coverage loss is the exact failure mode the tiered TTL is designed to prevent:

```
sum(rate(cycles_admin_audit_writes_total{outcome="error"}[5m])) > 0
```

**Semantic change for audit-log consumers.** Dashboards that assumed "audit entry exists ⇒ operation succeeded" must now check the entry's `status` or `error_code` field. Queries filtered by `status=201/200/204` return exactly the same rows as v0.1.25.19. See [Admin API guide](/admin-api/guide) for field semantics.

### Admin server Kubernetes probes

Like the runtime and events services, the admin server enables Spring Boot's liveness/readiness probes out of the box (`management.endpoint.health.probes.enabled=true`). In Kubernetes, wire probes to these paths:

```yaml
livenessProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 7979
readinessProbe:
  httpGet:
    path: /actuator/health/readiness
    port: 7979
```

### Admin authentication

The admin server uses two authentication schemes:

| Header | Variable | Purpose |
|---|---|---|
| `X-Admin-API-Key` | `ADMIN_API_KEY` | System-level and operator operations: tenant CRUD, API key management, audit logs, admin-only budget state, and runtime reservation admin-on-behalf-of list/detail/release |
| `X-Cycles-API-Key` | — | Tenant-scoped operations: budget ledgers, policies, reservations, balances, events, and tenant self-service webhooks |

For the full endpoint-to-header mapping with required permissions, see the [Architecture Overview — Authentication](/quickstart/architecture-overview-how-cycles-fits-together#authentication).

### Representative admin server configuration

```properties
# Server
server.port=7979
spring.application.name=cycles-admin-service

# Redis (same instance as cycles-server)
redis.host=${REDIS_HOST}
redis.port=${REDIS_PORT}
redis.password=${REDIS_PASSWORD}

# Admin key
admin.api-key=${ADMIN_API_KEY:}

# Webhook signing-secret encryption (required by default)
webhook.secret.encryption-key=${WEBHOOK_SECRET_ENCRYPTION_KEY:}
webhook.secret.allow-plaintext=${WEBHOOK_SECRET_ALLOW_PLAINTEXT:false}

# JSON
spring.jackson.serialization.write-dates-as-timestamps=false
spring.jackson.deserialization.fail-on-unknown-properties=false
spring.jackson.default-property-inclusion=non_null

# Logging
logging.level.root=INFO
logging.level.io.runcycles.admin=DEBUG

# Swagger
springdoc.api-docs.path=/api-docs
springdoc.api-docs.enabled=false
springdoc.swagger-ui.path=/swagger-ui.html
springdoc.swagger-ui.enabled=false

# Actuator
management.endpoints.web.exposure.include=health,info
management.endpoint.health.show-details=when-authorized
```

### Security note

The admin server exposes powerful management operations. In production:

- Run the admin server on an internal network not accessible to application traffic
- Use a strong, randomly generated `ADMIN_API_KEY`
- Keep Swagger UI and API docs disabled unless operators explicitly need them (`springdoc.swagger-ui.enabled=false`, `springdoc.api-docs.enabled=false`)

## Events Service Configuration

The events service (`cycles-server-events`) is an optional component for webhook delivery and CyclesEvidence signing.

### Ports (v0.1.25.9)

As of v0.1.25.9 the events service separates its application port from its management (actuator) port:

| Port | Default | Env Variable | Purpose |
|---|---|---|---|
| Application | `7980` | `SERVER_PORT` | Spring application port. The current reference service is an outbound worker and exposes no operator-facing HTTP API here. |
| Management | `9980` | `MANAGEMENT_PORT` | Actuator endpoints (`/actuator/health`, `/actuator/info`, `/actuator/prometheus`) |

**Migration from pre-.9:** Prometheus scrape configs must point to `:9980/actuator/prometheus`. Kubernetes liveness / readiness probes and Docker `HEALTHCHECK` must hit `:9980/actuator/health`. The published Docker image `HEALTHCHECK` has already been updated. No wire-format change for the dispatch surface.

Do not publish either port to the internet. Keep `9980` on an internal-only ClusterIP scraped by Prometheus; leave `7980` unexposed unless your deployment has an explicit internal control-plane use for that app port.

### Core config

| Variable | Default | Description |
|---|---|---|
| `REDIS_HOST` | localhost | Redis hostname (shared with admin/runtime) |
| `REDIS_PORT` | 6379 | Redis port |
| `REDIS_PASSWORD` | (empty) | Redis password |
| `REDIS_USERNAME` | (empty) | Redis ACL username |
| `REDIS_TLS_ENABLED` | `false` | Enable TLS for the Redis connection |
| `REDIS_CONNECT_TIMEOUT_MS` | `2000` | Redis connection timeout |
| `REDIS_SOCKET_TIMEOUT_MS` | `5000` | Redis non-blocking socket timeout |
| `REDIS_BLOCKING_SOCKET_TIMEOUT_MS` | `10000` | Redis timeout used by blocking queue operations |
| `WEBHOOK_SECRET_ENCRYPTION_KEY` | (empty; startup fails) | AES-256-GCM key for signing secret encryption. Base64, 32 bytes. Must match admin and runtime. Generate: `openssl rand -base64 32`. |
| `WEBHOOK_SECRET_ALLOW_PLAINTEXT` | `false` | Explicit local/development compatibility escape hatch. Never enable in production. |
| `EVIDENCE_SERVER_ID` | (empty) | Issuer base URL including `/v1`. Blank disables evidence signing and leaves pending evidence-source records untouched. Must match the runtime server when evidence is enabled. |
| `EVIDENCE_SIGNING_SIGNER_DID` | (empty) | Raw-hex Ed25519 public key. Must match the runtime server's public signer identity when evidence is enabled. |
| `EVIDENCE_SIGNING_PRIVATE_KEY_HEX` | (empty) | Raw-hex Ed25519 private key used to sign evidence envelopes. Secret; deploy only to `cycles-server-events`. |
| `EVIDENCE_ALLOW_EPHEMERAL_SIGNING_KEY` | `false` | Allow the worker to generate an ephemeral signing key when no keypair is configured. Development-only; leave `false` in production. |
| `EVIDENCE_STORE_BACKEND` | `redis` | Select the evidence-store bean. The reference service ships only `redis`; another value requires a custom `EvidenceStore` implementation in the application context. |
| `dispatch.pending.timeout-seconds` | 5 | BLMOVE blocking timeout (reliable-queue pattern) |
| `DISPATCH_LOOP_DELAY_MS` | `25` | Delay before the next dispatch-loop iteration |
| `DISPATCH_ORDERING_LEASE_MS` | `120000` | Global claim/send lease. Must cover queue claim, HTTP delivery, and Redis state-write time. |
| `DISPATCH_ORDERING_CONTENTION_BACKOFF_MS` | `500` | Backoff when another replica owns the ordering lease |
| `DISPATCH_PROCESSING_RECOVERY_IDLE_MS` | `180000` | Minimum idle age before an in-flight delivery is eligible for recovery |
| `DISPATCH_PROCESSING_RECOVERY_INTERVAL_MS` | `30000` | Interval between recovery passes |
| `DISPATCH_FAILED_MAX_LEN` | `10000` | Maximum malformed/untransitionable delivery IDs retained in the quarantine list |
| `DISPATCH_EVENT_OUTBOX_POLL_INTERVAL_MS` | `1000` | Lifecycle-event outbox polling interval |
| `DISPATCH_EVENT_OUTBOX_BATCH_SIZE` | `25` | Maximum outbox rows claimed per poll |
| `DISPATCH_EVENT_OUTBOX_CLAIM_LEASE_MS` | `30000` | Outbox claim lease |
| `DISPATCH_EVENT_OUTBOX_RETRY_DELAY_MS` | `5000` | Retry delay after an outbox publish failure |
| `DISPATCH_EVENT_OUTBOX_MAX_ATTEMPTS` | `100` | Maximum publish attempts before quarantine |
| `DISPATCH_EVENT_OUTBOX_FAILED_MAX_LEN` | `10000` | Maximum quarantined outbox rows retained |
| `dispatch.retry.poll-interval-ms` | 5000 | Retry queue poll interval (ms) |
| `dispatch.retry.batch-size` / `RETRY_BATCH_SIZE` | 100 | Max ready-for-retry deliveries processed per poll tick |
| `dispatch.http.timeout-seconds` | 30 | HTTP request timeout for webhook delivery |
| `dispatch.http.connect-timeout-seconds` | 5 | HTTP connect timeout |
| `WEBHOOK_URL_GUARD_ALLOW_PRIVATE_NETWORKS` | `false` | Development-only opt-out for the delivery-side private-network SSRF baseline; admin-configured blocked CIDRs remain enforced |
| `dispatch.max-delivery-age-ms` / `MAX_DELIVERY_AGE_MS` | 86400000 | Deliveries older than this auto-fail without further retries (24h). Also feeds `cycles_webhook_delivery_stale_total`. |
| `events.retention.event-ttl-days` / `EVENT_TTL_DAYS` | 90 | Redis TTL for event records |
| `events.retention.delivery-ttl-days` / `DELIVERY_TTL_DAYS` | 14 | Redis TTL for delivery records |
| `events.retention.cleanup-interval-ms` / `RETENTION_CLEANUP_INTERVAL_MS` | 3600000 | ZSET index cleanup interval (1h) |
| `RETENTION_LOCK_LEASE_MS` | `300000` | Distributed lease duration for retention cleanup |
| `SCHEDULING_POOL_SIZE` | `5` | Scheduler pool sized for dispatch, evidence, retry, recovery, and cleanup jobs |
| `cycles.metrics.tenant-tag.enabled` | `false` | Same toggle as the runtime, but the events service defaults to `false` (the runtime defaults to `true`). When `false`, `cycles_webhook_*` counters drop the `tenant` label to bound cardinality. |

### Evidence queue and store tuning

| Variable | Default | Description |
|---|---|---|
| `EVIDENCE_PENDING_KEY` | `evidence:pending` | Pending source-record list; must match the runtime server |
| `EVIDENCE_PROCESSING_KEY` | `evidence:processing` | In-flight reliable-queue list |
| `EVIDENCE_POP_TIMEOUT_SECONDS` | `5` | BLMOVE timeout |
| `EVIDENCE_LOOP_DELAY_MS` | `25` | Delay between worker iterations |
| `EVIDENCE_QUEUE_FAILURE_BACKOFF_MS` | `1000` | Backoff for record-level failures |
| `EVIDENCE_INFRASTRUCTURE_BACKOFF_MS` | `30000` | Backoff for signing/store infrastructure failures |
| `EVIDENCE_RECOVERY_IDLE_MS` | `120000` | Minimum idle age before in-flight recovery |
| `EVIDENCE_RECOVERY_INTERVAL_MS` | `30000` | Interval between evidence recovery passes |
| `EVIDENCE_RECOVERY_BATCH_SIZE` | `100` | Maximum in-flight records recovered per pass |
| `EVIDENCE_FAILED_KEY` | `evidence:failed` | Dead-letter list for deterministically malformed source records |
| `EVIDENCE_FAILED_MAX_LEN` | `10000` | Maximum retained dead-letter records |
| `EVIDENCE_STORE_KEY_PREFIX` | `evidence:envelope:` | Content-addressed Redis key prefix; must match the runtime server |
| `EVIDENCE_STORE_TTL_SECONDS` | `0` | Envelope TTL; `0` means no expiry |

### Per-subscription retry policy

Each subscription carries a `retry_policy` applied by the dispatcher's exponential-backoff loop in `DeliveryHandler`. Defaults (used when a subscription omits the field):

| Field | Default | Description |
|---|---|---|
| `max_retries` | 5 | Number of retry attempts before the delivery is marked failed. |
| `initial_delay_ms` | 1000 | First retry delay. Doubles with each attempt up to `max_delay_ms`. |
| `backoff_multiplier` | 2.0 | Exponential backoff factor. Delay for attempt *n* = `min(initial_delay_ms × multiplier^(n-1), max_delay_ms)`. |
| `max_delay_ms` | 60000 | Ceiling for the computed backoff delay. |

A delivery that exceeds `dispatch.max-delivery-age-ms` (default 24h) is failed immediately regardless of remaining retries.

### Events service metrics

Introduced in `cycles-server-events` v0.1.25.6 and expanded in later releases. The current service exposes 17 delivery, evidence, dispatcher, and security counters plus one delivery-latency timer. Metrics that carry `tenant` respect `cycles.metrics.tenant-tag.enabled`; metrics without that tag are unaffected. For the complete enumeration, see [Prometheus Metrics Reference](/how-to/prometheus-metrics-reference#events-service-cycles-server-events).

### Encryption key (shared across all services)

`WEBHOOK_SECRET_ENCRYPTION_KEY` must be the same on admin, runtime, and events services. Admin encrypts signing secrets on write; events decrypts on read. Current admin and events services fail startup when the key is missing. Local development may opt into plaintext explicitly with `WEBHOOK_SECRET_ALLOW_PLAINTEXT=true`; that escape hatch logs a warning and must never be enabled in production. Existing plaintext values remain readable during migration after a key is configured.

```bash
export WEBHOOK_SECRET_ENCRYPTION_KEY=$(openssl rand -base64 32)
```

### CyclesEvidence signer identity

To enable evidence, configure the same public identity on the runtime and events services:

- `EVIDENCE_SERVER_ID` — issuer URL, including `/v1`.
- `EVIDENCE_SIGNING_SIGNER_DID` — raw-hex public Ed25519 key.

Then configure only the events service with `EVIDENCE_SIGNING_PRIVATE_KEY_HEX`. Configure only the runtime server with `EVIDENCE_SIGNING_KID`, `EVIDENCE_SIGNING_NBF_MS`, and `EVIDENCE_SIGNING_RETIRED_KEYS` so it can publish `GET /v1/.well-known/cycles-jwks.json`.

### Representative events service configuration

```bash
# Required — must match admin and runtime servers
REDIS_HOST=redis.example.com
REDIS_PORT=6379
REDIS_PASSWORD=your-redis-password
WEBHOOK_SECRET_ENCRYPTION_KEY=$(openssl rand -base64 32)

# Optional — CyclesEvidence signer identity
EVIDENCE_SERVER_ID=https://cycles.example.com/v1
EVIDENCE_SIGNING_SIGNER_DID=b10554...c522
EVIDENCE_SIGNING_PRIVATE_KEY_HEX=4f9c...d20a

# Dispatch tuning
dispatch.pending.timeout-seconds=5
dispatch.retry.poll-interval-ms=5000
dispatch.http.timeout-seconds=30
dispatch.http.connect-timeout-seconds=5

# Delivery lifecycle
MAX_DELIVERY_AGE_MS=86400000       # 24h — deliveries older than this auto-fail

# Data retention
EVENT_TTL_DAYS=90                  # Event records in Redis
DELIVERY_TTL_DAYS=14               # Delivery records in Redis
RETENTION_CLEANUP_INTERVAL_MS=3600000  # ZSET index cleanup (1h)
```

See [Deploying the Events Service](/quickstart/deploying-the-events-service) for the full deployment guide.

## Next steps

- [Deploying the Full Cycles Stack](/quickstart/deploying-the-full-cycles-stack) — end-to-end deployment guide
- [Deploying the Events Service](/quickstart/deploying-the-events-service) — webhook delivery service setup
- [Self-Hosting the Cycles Server](/quickstart/self-hosting-the-cycles-server) — deployment guide
- [Architecture Overview](/quickstart/architecture-overview-how-cycles-fits-together) — system design
- [Client Configuration Reference](/configuration/client-configuration-reference-for-cycles-spring-boot-starter) — client-side properties



# SpEL Expression Reference for Cycles

The `@Cycles` annotation uses Spring Expression Language (SpEL) to evaluate `estimate` and `actual` cost values dynamically. This page is a comprehensive reference for writing SpEL expressions in Cycles.

## Where expressions are used

The `@Cycles` annotation accepts SpEL expressions in four places:

| Attribute | Evaluated when | Purpose |
|---|---|---|
| `value` / `estimate` | Before the method runs | Determines the reservation amount |
| `actual` | After the method returns | Determines the commit amount |
| `metadata` (since 0.2.5) | After the method returns | Produces the commit metadata map |
| Subject fields: `tenant`, `workspace`, `app`, `workflow`, `agent`, `toolset` (since 0.2.1) | Before the method runs | Resolves the subject field when the value starts with `#` |

### Subject-field expressions (since 0.2.1)

A subject attribute whose first non-whitespace character is `#` is evaluated as SpEL against the method invocation; any other value is treated as a literal:

```java
@Cycles(value = "1000", tenant = "#tenantId")
public String handle(String tenantId, String prompt) { ... }
```

Subject fields are evaluated before the guarded method runs, so `#result` is deliberately **not** available. Method parameters, `#args`, and `#target` are.

### Metadata expressions (since 0.2.5)

The `metadata` attribute is evaluated after the method returns and must yield a `Map<String, Object>`:

```java
@Cycles(value = "1000", metadata = "{'app_request_id': #requestId, 'model': #result.model}")
public LlmResponse call(String requestId, String prompt) { ... }
```

In addition to method parameters, `#args`, `#target`, and `#result`, metadata expressions expose `#method` and a root object with `target`, `args`, `result`, and `method` properties. The evaluated map is merged with programmatic `CyclesContextHolder` commit metadata; programmatic metadata wins on key conflicts.

## Available variables

### Method parameters

Parameters are available by index and by name:

```java
@Cycles("#p0 * 10")
public String generate(int tokens) { ... }
```

| Variable | Meaning |
|---|---|
| `#p0`, `#p1`, `#p2`, ... | Parameters by index (zero-based) |
| `#paramName` | Parameters by name (requires `-parameters` compiler flag) |

#### Parameter names

To use parameter names instead of indexes, compile with the `-parameters` flag:

```xml
<!-- Maven -->
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-compiler-plugin</artifactId>
  <configuration>
    <parameters>true</parameters>
  </configuration>
</plugin>
```

```groovy
// Gradle
tasks.withType(JavaCompile) {
    options.compilerArgs << '-parameters'
}
```

With this flag:

```java
@Cycles("#tokens * 10")
public String generate(int tokens) { ... }
```

Without it, use `#p0`:

```java
@Cycles("#p0 * 10")
public String generate(int tokens) { ... }
```

### Return value

The `#result` variable is available in the `actual` and `metadata` expressions, evaluated after the method returns:

```java
@Cycles(estimate = "5000", actual = "#result.usage.totalTokens * 8")
public ChatResponse chat(String prompt) { ... }
```

If the method returns `null`, `#result` is `null`. Accessing a property on it throws a `SpelEvaluationException` (EL1007/EL1011 — property or method access on a null context object), and an expression that evaluates to `null` overall causes the starter to throw `IllegalArgumentException` (`"Expression evaluated to null: ..."`).

### Other variables

| Variable | Meaning |
|---|---|
| `#args` | All method arguments as an `Object[]` array |
| `#target` | The target object instance (the bean the method belongs to) |

## Expression examples

### Fixed values

```java
@Cycles("500")
public String summarize(String text) { ... }
```

A literal number is the simplest expression. It evaluates to that value every time.

### Arithmetic on parameters

```java
@Cycles("#p0 * 10")
public String generate(int maxTokens) { ... }
```

```java
@Cycles("#p0.length() / 4 * 8")
public String processText(String input) { ... }
```

### Using named parameters

```java
@Cycles("#maxTokens * 10")
public String generate(int maxTokens) { ... }
```

### Using the return value

```java
@Cycles(estimate = "5000", actual = "#result.length() * 5")
public String translate(String text) { ... }
```

```java
@Cycles(estimate = "#p1 * 10",
        actual = "#result.usage.totalTokens * 8")
public ChatResponse chat(String prompt, int estimatedTokens) { ... }
```

### Accessing nested properties

```java
@Cycles(estimate = "#request.estimatedTokens * 10",
        actual = "#result.metadata.totalCost")
public Response process(Request request) { ... }
```

### Conditional expressions

```java
@Cycles("#p0.length() > 1000 ? 10000 : 2000")
public String summarize(String text) { ... }
```

### Math functions

```java
@Cycles("T(Math).max(#p0 * 10, 1000)")
public String generate(int tokens) { ... }
```

```java
@Cycles("T(Math).min(#p0.length() / 4 * 8, 50000)")
public String process(String input) { ... }
```

### Accessing the args array

```java
@Cycles("#args[0].length() * #args[1]")
public String process(String text, int costPerChar) { ... }
```

### Accessing the target bean

```java
@Cycles("#target.getEstimateMultiplier() * #p0")
public String process(int tokens) {
    // ...
}

public int getEstimateMultiplier() {
    return 10;
}
```

## Evaluation rules

### Return type

The expression must evaluate to a `Number`. The result is converted to a `long` via `Number.longValue()`.

### Non-negative

The evaluated value must be >= 0. A negative value throws `IllegalArgumentException`.

### Null safety

If the expression evaluates to `null`, an `IllegalArgumentException` is thrown. Guard against null:

```java
// Safe: use a fallback
@Cycles(actual = "#result != null ? #result.cost : 0")
public Result process(String input) { ... }
```

### Estimate vs actual

| Attribute | `#result` available? | When evaluated |
|---|---|---|
| `value` / `estimate` | No | Before method execution |
| `actual` | Yes | After method returns |
| `metadata` | Yes | After method returns |
| Subject fields | No | Before method execution |

If `actual` is not specified and `useEstimateIfActualNotProvided` is `true` (the default), the estimate value is used as the actual at commit time.

## Common patterns

### Token-based estimation

```java
@Cycles(estimate = "#prompt.length() / 4 * 10",
        actual = "#result.usage.totalTokens * 10",
        unit = "USD_MICROCENTS")
public ChatResponse complete(String prompt) { ... }
```

### Fixed estimate with actual from response

```java
@Cycles(estimate = "10000",
        actual = "#result.cost",
        unit = "USD_MICROCENTS")
public ApiResponse callExternalApi(Request request) { ... }
```

### Multiple parameters

```java
@Cycles("#p0 * #p1 * 8")
public String batchProcess(int documents, int tokensPerDoc) { ... }
```

### Using enum or constant values via SpEL

```java
@Cycles("T(com.example.CostTable).estimateFor(#p0)")
public String process(String modelName) { ... }
```

## Troubleshooting

### "Expression evaluated to null"

The expression returned null. Common cause: accessing a property on a null object. Add a null check:

```java
actual = "#result?.cost != null ? #result.cost : 0"
```

### "Charge amount must not be negative"

The expression evaluated to a negative number. Ensure your math cannot produce negative values:

```java
estimate = "T(Math).max(#p0 * 10, 0)"
```

### "Parameter name not found"

You used `#paramName` but did not compile with `-parameters`. Use `#p0` index-based access or add the compiler flag.

### Expression parse errors

Check for typos in method names, property paths, or operator usage. SpEL follows Java-like syntax but uses `#` for variables.

## Next steps

- [Getting Started with the Spring Boot Starter](/quickstart/getting-started-with-the-cycles-spring-boot-starter) — annotation usage
- [Client Configuration Reference](/configuration/client-configuration-reference-for-cycles-spring-boot-starter) — all configuration properties
- [Error Handling Patterns](/how-to/error-handling-patterns-in-cycles-client-code) — handling evaluation failures



# Spring AI Starter Configuration Reference

This page is the complete configuration reference for `io.runcycles:cycles-spring-ai-starter` 0.4.0. The authoritative implementation is [`CyclesSpringAiProperties`](https://github.com/runcycles/cycles-spring-ai-starter/blob/main/cycles-spring-ai-starter/src/main/java/io/runcycles/client/java/springai/autoconfigure/CyclesSpringAiProperties.java).

The integration uses two property namespaces:

- `cycles.*` configures the underlying `cycles-client-java-spring` client, including the server URL, API key, subject defaults, HTTP timeouts, and retries.
- `cycles.spring-ai.*` configures the Spring AI advisors, token accounting, tool labels, failure behavior, and tracing.

See [Spring Client Configuration](/configuration/client-configuration-reference-for-cycles-spring-boot-starter) for the underlying `cycles.*` namespace.

## Minimal configuration

```yaml
cycles:
  base-url: http://localhost:7878
  api-key: ${CYCLES_API_KEY}
  tenant: acme
  workspace: production
  app: support-agent

  spring-ai:
    enabled: true
    default-estimate: 1000
    estimate-unit: TOKENS
```

The starter auto-attaches both its non-streaming and streaming advisors to every `ChatClient` built from Spring Boot's auto-configured `ChatClient.Builder`.

It does not automatically gate:

- a manually constructed `ChatClient.Builder` that did not receive Spring Boot's customizers;
- raw provider SDK calls;
- tool callbacks, unless you explicitly wrap them with `CyclesToolGate`;
- a `ChatClient` whose call path is already wrapped by the `@Cycles` Spring Boot starter.

Wrapping the same call with both the Spring AI advisor and `@Cycles` creates two reservations. Choose one gate for each call path.

## Property reference

| Property | Type | Default | Behavior |
|---|---|---|---|
| `cycles.spring-ai.enabled` | Boolean | `true` | Master switch. When `false`, the Spring AI auto-configuration does not register its beans. |
| `cycles.spring-ai.default-estimate` | Long | `1000` | Pre-call reservation amount when prompt-derived estimation is disabled or unavailable. Must be non-negative; invalid values fail property binding at startup. |
| `cycles.spring-ai.estimate-unit` | String | `USD_MICROCENTS` | Unit for estimates and commits: `USD_MICROCENTS`, `TOKENS`, `CREDITS`, or `RISK_POINTS`. It must match the target budget ledger's unit. |
| `cycles.spring-ai.action-kind` | String | `llm.chat` | `action.kind` recorded on chat reservations. |
| `cycles.spring-ai.action-name` | String | `spring-ai-chat` | `action.name` recorded on chat reservations. |
| `cycles.spring-ai.tool-action-kind` | String | `tool.call` | `action.kind` used by `CyclesToolCallback`-wrapped tools. |
| `cycles.spring-ai.tool-action-name-prefix` | String | `spring-ai-tool:` | Prefix joined with the wrapped tool name, such as `spring-ai-tool:get_weather`. |
| `cycles.spring-ai.fail-open` | Boolean | `false` | When `true`, reservation or commit transport/HTTP failures are logged and the model call proceeds. Explicit Cycles budget denials are always surfaced. |
| `cycles.spring-ai.input-cost-per-token` | Long | `0` | Cost per prompt token in `estimate-unit`. Must be non-negative. Used for actual-cost commits when the provider returns token breakdowns. |
| `cycles.spring-ai.output-cost-per-token` | Long | `0` | Cost per completion token in `estimate-unit`. Must be non-negative. Used for actual-cost commits when the provider returns token breakdowns. |
| `cycles.spring-ai.estimate-from-prompt` | Boolean | `false` | Derives the reservation from estimated prompt tokens when at least one token rate is positive. Falls back to `default-estimate` when derivation is unavailable or returns zero. |
| `cycles.spring-ai.token-estimator-encoding` | String | unset | Selects a jtokkit BPE encoding when jtokkit is present. Supported names: `cl100k_base`, `o200k_base`, `p50k_base`, `p50k_edit`, and `r50k_base`. |
| `cycles.spring-ai.emit-reservation-id-on-trace` | Boolean | `true` | Adds `cycles.reservation_id` as a high-cardinality trace value when the Cycles observation convention is explicitly attached. |

Spring Boot relaxed binding also accepts uppercase environment-variable forms, for example:

```bash
CYCLES_SPRING_AI_ENABLED=true
CYCLES_SPRING_AI_DEFAULT_ESTIMATE=1000
CYCLES_SPRING_AI_ESTIMATE_UNIT=TOKENS
CYCLES_SPRING_AI_FAIL_OPEN=false
```

## Actual usage calculation

The advisor chooses the committed amount in this order:

1. With `estimate-unit=TOKENS`, it commits `Usage.getTotalTokens()` when the provider supplies it.
2. With either token rate configured, it commits `promptTokens × inputRate + completionTokens × outputRate`.
3. Otherwise, it commits `default-estimate`.

If both prompt and completion token counts are missing, the advisor commits the estimate instead of treating missing usage as zero. If only one breakdown is present, it charges the available side and treats the missing side as zero.

### USD microcents example

`USD_MICROCENTS` uses 100,000,000 units per US dollar. A model priced at $2.50 per million input tokens and $10.00 per million output tokens therefore uses rates of 250 and 1,000:

```yaml
cycles:
  spring-ai:
    estimate-unit: USD_MICROCENTS
    default-estimate: 125000
    input-cost-per-token: 250
    output-cost-per-token: 1000
```

Keep model prices in application configuration and update them when the provider's pricing changes. The starter does not fetch provider pricing.

## Prompt-derived reservation sizing

Enable prompt sizing only when at least one token rate is positive:

```yaml
cycles:
  spring-ai:
    estimate-from-prompt: true
    input-cost-per-token: 250
    output-cost-per-token: 1000
```

The default `CharsPerTokenEstimator` approximates prompt tokens as characters divided by four. It then multiplies the estimated input-token count by the sum of the input and output rates, assuming output length is comparable to input length.

For OpenAI-family BPE estimation, add jtokkit:

```xml
<dependency>
  <groupId>com.knuddels</groupId>
  <artifactId>jtokkit</artifactId>
  <version>1.1.0</version>
</dependency>
```

Then select an encoding:

```yaml
cycles:
  spring-ai:
    estimate-from-prompt: true
    token-estimator-encoding: o200k_base
```

The jtokkit dependency is optional and is not pulled transitively. If an encoding is configured without jtokkit on the classpath, startup logs a warning and the starter uses the characters-per-four estimator.

## Failure behavior

The default is fail closed:

```yaml
cycles:
  spring-ai:
    fail-open: false
```

| Failure | `fail-open=false` | `fail-open=true` |
|---|---|---|
| Explicit `DENY` from Cycles | Throws `CyclesBudgetDeniedException` | Throws `CyclesBudgetDeniedException` |
| Reservation transport or non-2xx failure | Fails the call | Logs and calls the model without a reservation |
| Malformed successful reservation response | Fails the call | Logs and calls the model without a reservation |
| Commit transport or non-2xx failure | Fails after model execution | Logs and returns the model result |
| Model call failure | Releases the reservation best-effort, then propagates the model error | Same |
| Stream error or cancellation | Releases the reservation best-effort | Same |

Release failures are logged and left for reservation TTL recovery. `fail-open` is an availability tradeoff, not an observe-only mode: when it allows a call after a Cycles failure, that call is not protected by a live reservation.

## Per-call subject routing

The default `PropertiesSubjectResolver` uses the underlying `cycles.tenant`, `cycles.workspace`, `cycles.app`, `cycles.workflow`, `cycles.agent`, and `cycles.toolset` values.

For multi-tenant applications, provide a `SubjectResolver` bean:

```java
@Bean
SubjectResolver authenticatedSubjectResolver(CyclesProperties defaults) {
    return request -> {
        Authentication auth = SecurityContextHolder.getContext().getAuthentication();
        String tenant = auth != null && auth.isAuthenticated()
                ? auth.getName()
                : defaults.getTenant();

        return Subject.builder()
                .tenant(tenant)
                .workspace(defaults.getWorkspace())
                .app(defaults.getApp())
                .build();
    };
}
```

The resolver receives the current `ChatClientRequest` for chat calls. Tool-gating calls pass `null`, because Spring AI tool callbacks do not carry that request; custom resolvers must handle that path.

## Custom token estimator

A custom `PromptTokenEstimator` bean replaces both the characters-per-four and jtokkit defaults:

```java
@Bean
PromptTokenEstimator providerTokenEstimator(ProviderTokenizer tokenizer) {
    return request -> request.prompt().getInstructions().stream()
            .map(Message::getText)
            .filter(Objects::nonNull)
            .mapToLong(tokenizer::countTokens)
            .sum();
}
```

The custom estimator affects pre-call reservation sizing only. Actual commits still use the provider's `ChatResponse.Usage` when available.

## Tool gating

Tool gating is explicit:

```java
@Bean
ToolCallback getWeatherTool(CyclesToolGate gate, WeatherTool weatherTool) {
    ToolCallback raw = ToolCallbacks.from(weatherTool)[0];
    return gate.wrap(raw);
}
```

Only wrapped callbacks reserve and settle through Cycles. The tool gate uses `tool-action-kind` and `tool-action-name-prefix`, and commits the configured default estimate because Spring AI tool callbacks do not expose a standard actual-usage result to the gate.

If the tool internally calls an auto-configured `ChatClient`, that nested model call is separately gated by the chat advisor. A raw provider SDK or a hand-built `ChatClient` that bypasses the auto-configured builder is not.

## Trace correlation

`CyclesChatClientObservationConvention` adds these low-cardinality tags:

- `cycles.tenant`
- `cycles.workspace`
- `cycles.app`
- `cycles.action_kind`
- `cycles.action_name`

It also adds `cycles.reservation_id` as a high-cardinality value by default. The convention is registered as a bean but is not attached automatically:

```java
@Bean
ChatClient governedChatClient(
        ChatClient.Builder builder,
        CyclesChatClientObservationConvention cyclesConvention) {
    return builder
            .observationConvention(cyclesConvention)
            .build();
}
```

Disable only the reservation ID when high-cardinality trace values are too expensive:

```yaml
cycles:
  spring-ai:
    emit-reservation-id-on-trace: false
```

## Auto-configuration conditions and overrides

The integration activates when all of these conditions hold:

- Spring AI `ChatClient` and `ChatClientCustomizer` are on the classpath;
- the underlying starter has created a `CyclesClient` bean;
- `cycles.spring-ai.enabled` is `true` or absent.

User-provided beans replace the defaults for:

- `SubjectResolver`
- `PromptTokenEstimator`
- `CyclesBudgetAdvisor`
- `CyclesBudgetStreamAdvisor`
- `CyclesToolGate`
- `CyclesChatClientObservationConvention`

To replace only advisor attachment while keeping other `ChatClientCustomizer` beans, provide a bean named `cyclesChatClientCustomizer`.

## Related

- [Integrating Cycles with Spring AI](/how-to/integrating-cycles-with-spring-ai) — installation and complete examples
- [Spring Client Configuration](/configuration/client-configuration-reference-for-cycles-spring-boot-starter) — underlying HTTP client, subject, timeout, and retry properties
- [Spring AI starter source](https://github.com/runcycles/cycles-spring-ai-starter)



# TypeScript Client Configuration Reference

This is the complete reference for all configuration options available in the `runcycles` TypeScript client.

## CyclesConfig

All configuration is provided through the `CyclesConfig` constructor.

### Required fields

| Field | Type | Description |
|---|---|---|
| `baseUrl` | `string` | Base URL of the Cycles server (e.g., `http://localhost:7878`) |
| `apiKey` | `string` | API key for authentication |

### Subject defaults

These fields set default values for the Subject used in `withCycles` calls. They apply to all guarded functions unless overridden at the HOF level.

| Field | Type | Default | Description |
|---|---|---|---|
| `tenant` | `string \| undefined` | `undefined` | Default tenant |
| `workspace` | `string \| undefined` | `undefined` | Default workspace |
| `app` | `string \| undefined` | `undefined` | Default application name |
| `workflow` | `string \| undefined` | `undefined` | Default workflow |
| `agent` | `string \| undefined` | `undefined` | Default agent |
| `toolset` | `string \| undefined` | `undefined` | Default toolset |

### HTTP timeouts

| Field | Type | Default | Description |
|---|---|---|---|
| `connectTimeout` | `number` | `2000` | Connection timeout in milliseconds. Summed with `readTimeout` (see note below). |
| `readTimeout` | `number` | `5000` | Read timeout in milliseconds. Summed with `connectTimeout` (see note below). |

::: warning Timeout behavior
Node's built-in `fetch` does not distinguish connection timeout from read timeout. `connectTimeout` and `readTimeout` are **summed into a single `AbortSignal.timeout()`** value (default: 2000 + 5000 = **7000ms total**) that caps the entire request duration. If you need a 5-second maximum, set values like `connectTimeout: 2000, readTimeout: 3000`.
:::

### Retry configuration

Controls the commit retry engine and bounded drain.

| Field | Type | Default | Description |
|---|---|---|---|
| `retryEnabled` | `boolean` | `true` | Enable automatic commit retries |
| `retryMaxAttempts` | `number` | `5` | Maximum number of retry attempts |
| `retryInitialDelay` | `number` | `500` | Delay before the first retry (milliseconds) |
| `retryMultiplier` | `number` | `2.0` | Backoff multiplier between retries |
| `retryMaxDelay` | `number` | `30000` | Maximum delay between retries (milliseconds) |
| `retryFlushTimeout` | `number` | `10000` | Default bound used by `flushPendingCommits()` (milliseconds); `0` disables waiting |

#### How retry works

When settlement fails transiently, the retry engine schedules a same-key retry using exponential backoff:

```
Attempt 1: wait 500ms
Attempt 2: wait 1000ms
Attempt 3: wait 2000ms
Attempt 4: wait 4000ms
Attempt 5: wait 8000ms
```

With the default settings the backoff never reaches the cap; `retryMaxDelay` only kicks in once the exponential delay would exceed 30000ms (for example, with a higher `retryMaxAttempts`).

HTTP 429 honors a valid `Retry-After` floor, capped by the SDK's bounded-delay policy. Authentication failures and unclassifiable 4xx responses stop the current retry run but retain the durable record. A genuine, understood client rejection stops retrying and removes the record.

Retries run in the background, but they are not memory-only: known actual usage is journaled before the first settlement request.

### Durable journal

| Field | Type | Default | Description |
|---|---|---|---|
| `journalEnabled` | `boolean` | `true` | Persist unresolved known-actual settlement across process restarts |
| `journalDir` | `string \| undefined` | `undefined` | Journal base directory; `undefined` uses `~/.runcycles/commit-journal` |

A schema-valid HTTP `200` commit or schema-valid HTTP `201` event proves success. Ambiguous outcomes, retry exhaustion, authentication failures, and unclassifiable 4xx responses remain journaled for replay. An expired commit switches to `POST /v1/events` with the original idempotency key.

The journal is partitioned by server and principal. Configure `tenant` so pending records remain discoverable after API-key rotation. Journal records do not store API keys, but they contain settlement bodies and metadata; protect the directory as sensitive application state.

Call `flushPendingCommits(timeoutMs?)` during graceful shutdown. The timeout is process-wide across registered retry engines; unfinished records stay on disk for the next run.

## Programmatic configuration

```typescript
import { CyclesConfig } from "runcycles";

const config = new CyclesConfig({
  // Required
  baseUrl: "http://localhost:7878",
  apiKey: "cyc_live_...",

  // Subject defaults
  tenant: "acme",
  workspace: "production",
  app: "support-bot",

  // HTTP settings (milliseconds)
  connectTimeout: 2000,
  readTimeout: 5000,

  // Commit retry
  retryEnabled: true,
  retryMaxAttempts: 5,
  retryInitialDelay: 500,
  retryMultiplier: 2.0,
  retryMaxDelay: 30000,
  retryFlushTimeout: 10000,

  // Durable settlement journal
  journalEnabled: true,
  journalDir: undefined, // ~/.runcycles/commit-journal
});
```

## Environment variable configuration

Use `CyclesConfig.fromEnv()` to load configuration from environment variables. The default prefix is `CYCLES_`:

```typescript
const config = CyclesConfig.fromEnv();
```

| Environment variable | Maps to | Required |
|---|---|---|
| `CYCLES_BASE_URL` | `baseUrl` | Yes |
| `CYCLES_API_KEY` | `apiKey` | Yes |
| `CYCLES_TENANT` | `tenant` | No |
| `CYCLES_WORKSPACE` | `workspace` | No |
| `CYCLES_APP` | `app` | No |
| `CYCLES_WORKFLOW` | `workflow` | No |
| `CYCLES_AGENT` | `agent` | No |
| `CYCLES_TOOLSET` | `toolset` | No |
| `CYCLES_CONNECT_TIMEOUT` | `connectTimeout` | No |
| `CYCLES_READ_TIMEOUT` | `readTimeout` | No |
| `CYCLES_RETRY_ENABLED` | `retryEnabled` | No |
| `CYCLES_RETRY_MAX_ATTEMPTS` | `retryMaxAttempts` | No |
| `CYCLES_RETRY_INITIAL_DELAY` | `retryInitialDelay` | No |
| `CYCLES_RETRY_MULTIPLIER` | `retryMultiplier` | No |
| `CYCLES_RETRY_MAX_DELAY` | `retryMaxDelay` | No |
| `CYCLES_RETRY_FLUSH_TIMEOUT` | `retryFlushTimeout` | No |
| `CYCLES_JOURNAL_ENABLED` | `journalEnabled` | No |
| `CYCLES_JOURNAL_DIR` | `journalDir` | No |

A custom prefix can be passed: `CyclesConfig.fromEnv("MY_PREFIX_")` reads `MY_PREFIX_BASE_URL`, `MY_PREFIX_API_KEY`, etc.

## `withCycles` options

The `withCycles` HOF accepts an options object that controls reservation behavior per-call. These are separate from the `CyclesConfig` connection settings above. For full documentation and examples, see [Getting Started with the TypeScript Client — withCycles parameters](/quickstart/getting-started-with-the-typescript-client#withcycles-parameters).

| Parameter | Type | Default | Description |
|---|---|---|---|
| `estimate` | `number \| Function` | (required) | Estimated cost. Number constant or function receiving the wrapped function's arguments. |
| `actual` | `number \| Function \| undefined` | `undefined` | Actual cost. Number constant or function receiving the return value. Defaults to estimate. |
| `actionKind` | `string \| ((...args) => string \| undefined)` | `"unknown"` | Action category (e.g. `"llm.completion"`). |
| `actionName` | `string \| ((...args) => string \| undefined)` | `"unknown"` | Action identifier (e.g. `"gpt-4"`). |
| `actionTags` | `string[] \| undefined` | `undefined` | Tags for filtering and reporting. |
| `unit` | `string` | `"USD_MICROCENTS"` | Budget unit: `"USD_MICROCENTS"`, `"TOKENS"`, `"CREDITS"`, `"RISK_POINTS"`. |
| `ttlMs` | `number` | `60000` | Reservation TTL in milliseconds (range: 1000–86400000). |
| `gracePeriodMs` | `number \| undefined` | `undefined` | Grace period after TTL expiry in milliseconds. When `undefined`, the server applies its default (5000ms). Valid range: 0–60,000. |
| `overagePolicy` | `string` | `"ALLOW_IF_AVAILABLE"` | `"REJECT"`, `"ALLOW_IF_AVAILABLE"`, or `"ALLOW_WITH_OVERDRAFT"`. |
| `dryRun` | `boolean` | `false` | If `true`, evaluate without persisting. Function does not execute. |
| `tenant` | `string \| ((...args) => string \| undefined)` | `undefined` | Subject tenant override (takes precedence over config default). |
| `workspace` | `string \| ((...args) => string \| undefined)` | `undefined` | Subject workspace override. |
| `app` | `string \| ((...args) => string \| undefined)` | `undefined` | Subject app override. |
| `workflow` | `string \| ((...args) => string \| undefined)` | `undefined` | Subject workflow override. |
| `agent` | `string \| ((...args) => string \| undefined)` | `undefined` | Subject agent override. |
| `toolset` | `string \| ((...args) => string \| undefined)` | `undefined` | Subject toolset override. |
| `dimensions` | `Record<string, string> \| undefined` | `undefined` | Custom dimensions for the subject. |
| `client` | `CyclesClient \| undefined` | `undefined` | Explicit client. Falls back to module-level default. |
| `useEstimateIfActualNotProvided` | `boolean` | `true` | If `true` and `actual` is not set, use estimate as actual at commit. |

Since 0.3.0, `actionKind`, `actionName`, and the six subject fields accept a callable that receives the wrapped function's arguments and is resolved per call. If the callable returns `undefined`, the field falls back to the `CyclesConfig` default for subject fields, or to `"unknown"` for `actionKind`/`actionName`.

## Setting a default client

Instead of passing `client` to every `withCycles` call, set a module-level default:

```typescript
import { CyclesClient, CyclesConfig, setDefaultClient, setDefaultConfig } from "runcycles";

// Option 1: Set a config (client created lazily on first invocation)
setDefaultConfig(new CyclesConfig({
  baseUrl: "http://localhost:7878",
  apiKey: "cyc_live_...",
  tenant: "acme",
}));

// Option 2: Set an explicit client
setDefaultClient(new CyclesClient(new CyclesConfig({
  baseUrl: "http://localhost:7878",
  apiKey: "cyc_live_...",
})));
```

Client resolution is deferred to the first invocation and then cached — the wrapper binds permanently to the resolved client after its first call. A later `setDefaultClient()` call will not affect already-invoked wrappers.

## Resolution order

For each Subject field, the HOF resolves the value using this priority:

1. **HOF parameter** — if set in the `withCycles` options, it wins
2. **Config default** — if set on the `CyclesConfig` instance

If neither provides a value, the field is omitted from the request.

## Disabling retry

```typescript
const config = new CyclesConfig({
  baseUrl: "http://localhost:7878",
  apiKey: "cyc_live_...",
  retryEnabled: false,
});
```

This disables active background retries, not durability. Failed known-actual settlement remains journaled for replay on a later run while `journalEnabled` is `true`. Disable both only when the application supplies equivalent durable settlement recovery.

## Aggressive retry for critical commits

```typescript
const config = new CyclesConfig({
  baseUrl: "http://localhost:7878",
  apiKey: "cyc_live_...",
  retryMaxAttempts: 10,
  retryInitialDelay: 200,
  retryMultiplier: 1.5,
  retryMaxDelay: 60000,
});
```

## Next steps

- [Getting Started with the TypeScript Client](/quickstart/getting-started-with-the-typescript-client) — quick start guide
- [Error Handling Patterns](/how-to/error-handling-patterns-in-cycles-client-code) — error handling patterns
- [Using the Client Programmatically](/how-to/using-the-cycles-client-programmatically) — direct client usage
- [SDK Settlement Recovery and Durability](/protocol/sdk-settlement-recovery-and-durability) — journal, replay, expiry fallback, and guarantee boundary
- [Server Configuration Reference](/configuration/server-configuration-reference-for-cycles) — server-side properties