Webhooks and Events

Cycles emits events for every observable state change — budget exhaustion, reservation denials, tenant lifecycle changes, API key operations, and system health. Webhooks deliver these events to external endpoints via HTTP POST with HMAC-SHA256 signatures.

Core Concepts

Events

An event is an immutable record of a state change. Every event has:

• event_type — dotted format like budget.exhausted or reservation.denied
• category — one of: budget, reservation, tenant, api_key, policy, webhook, system
• tenant_id — which tenant is affected
• source — which service emitted it (cycles-server, cycles-admin, expiry-sweeper)
• data — event-specific payload (varies by type)

Events are stored in Redis with a 90-day TTL (configurable).

Webhook Subscriptions

A subscription defines which events to deliver and where:

• url — HTTPS endpoint to receive HTTP POST requests
• event_types — specific events to receive (e.g., ["budget.exhausted", "reservation.denied"])
• event_categories — receive all events in a category (additive with event_types)
• signing_secret — HMAC-SHA256 key for payload verification

Delivery Semantics

• At-least-once — events may be delivered more than once. Deduplicate using event_id.
• Ordered within tenant — events for the same tenant are dispatched in order.
• Non-blocking — webhook delivery never blocks the API operation that produced the event.
• Retry with backoff — failed deliveries retry with exponential backoff (default: 5 retries).
• Auto-disable — subscriptions are disabled after consecutive failures (default: 10).

Architecture

Admin ServerCRUD operations

Runtime Serverreserve / commit

event:{id} + delivery:{id} + LPUSH dispatch:pending

Redisdispatch:pending queue

BRPOP

Cycles Events ServicePort 7980

HTTP POST + X-Cycles-Signature

Your Webhook Endpoint

Event flow architecture: 1. Both the Admin Server (CRUD operations) and Runtime Server (reserve/commit) write events to Redis. Each event creates an event:{id} record, a delivery:{id} record, and is pushed to the dispatch:pending queue via LPUSH. 2. Redis holds the dispatch:pending queue. 3. The Cycles Events Service (port 7980) consumes events via BRPOP from the dispatch:pending queue. 4. The Events Service delivers webhooks via HTTP POST with X-Cycles-Signature header to your webhook endpoint. The events service is optional. If not deployed, events accumulate in Redis with TTL and are delivered when the service starts.

The events service is optional. If not deployed, events accumulate in Redis with TTL and are delivered when the service starts.

47 Registered Event Types

Category	Count	Examples
budget	16	budget.exhausted, budget.threshold_crossed, budget.over_limit_entered, budget.funded, budget.reset_spent
reservation	5	reservation.denied, reservation.commit_overage, reservation.expired
tenant	6	tenant.created, tenant.suspended, tenant.closed
api_key	6	api_key.created, api_key.revoked, api_key.auth_failed
policy	3	policy.created, policy.updated, policy.deleted
webhook	6	webhook.created, webhook.paused, webhook.disabled
system	5	system.store_connection_lost, system.webhook_delivery_failed

Tenant Self-Service

Tenants can create their own webhook subscriptions via /v1/webhooks (requires webhooks:write permission). Tenant webhooks are restricted to budget, reservation, and tenant events: 27 of the 47 registered event types — plus the additive _via_tenant_cascade fan-out events that the reference admin server emits in those same categories during a tenant close (see Tenant-Close Cascade Semantics). API key, policy, webhook lifecycle, and system events are admin-only.

Security

• HMAC-SHA256 — every delivery includes X-Cycles-Signature: sha256=<hex> for payload verification
• Encryption at rest — signing secrets encrypted in Redis with AES-256-GCM
• SSRF prevention — private IP ranges blocked by default, HTTPS required in production

Learn More

• Webhook Integrations Guide — PagerDuty, Slack, ServiceNow examples with code
• Security Hardening — webhook URL security and secret rotation
• Production Operations — events service deployment and failure handling