Webhook Integrations

Cycles emits webhook events for every state change — budget exhaustion, reservation denials, API key revocations, tenant lifecycle changes, and more. This guide shows concrete examples of webhook payloads and how to integrate with common services.

Webhook Payload Examples

reservation.denied

Emitted when a reservation or decide request is denied (budget exceeded, overdraft limit, etc.).

{
  "event_id": "evt_a1b2c3d4e5f67890",
  "event_type": "reservation.denied",
  "category": "reservation",
  "timestamp": "2026-04-01T14:32:01.456Z",
  "tenant_id": "acme-corp",
  "scope": "tenant:acme-corp/workspace:prod/agent:support-bot",
  "actor": {
    "type": "api_key",
    "key_id": "key_9f8e7d6c-5b4a-3210",
    "source_ip": "10.0.1.42"
  },
  "source": "cycles-server",
  "data": {
    "scope": "tenant:acme-corp/workspace:prod/agent:support-bot",
    "unit": "USD_MICROCENTS",
    "reason_code": "BUDGET_EXCEEDED",
    "requested_amount": 5000000,
    "remaining": 0
  },
  "request_id": "req_abc123"
}

budget.exhausted

Emitted when remaining budget hits zero.

{
  "event_id": "evt_f0e1d2c3b4a59687",
  "event_type": "budget.exhausted",
  "category": "budget",
  "timestamp": "2026-04-01T14:32:00.123Z",
  "tenant_id": "acme-corp",
  "scope": "tenant:acme-corp/workspace:prod",
  "actor": {
    "type": "api_key",
    "key_id": "key_9f8e7d6c-5b4a-3210"
  },
  "source": "cycles-server",
  "data": {
    "scope": "tenant:acme-corp/workspace:prod",
    "unit": "USD_MICROCENTS",
    "threshold": 1.0,
    "utilization": 1.0,
    "allocated": 100000000,
    "remaining": 0,
    "spent": 85000000,
    "reserved": 15000000,
    "direction": "rising"
  }
}

budget.threshold_crossed

Emitted when utilization crosses a configured threshold. Default thresholds if not specified on the subscription: 80%, 95%, and 100% (via WebhookThresholdConfig.budget_utilization). The direction field is "rising" when utilization increases past the threshold and "falling" when it drops back below, preventing duplicate alerts.

{
  "event_id": "evt_1122334455667788",
  "event_type": "budget.threshold_crossed",
  "category": "budget",
  "timestamp": "2026-04-01T13:15:00.789Z",
  "tenant_id": "acme-corp",
  "scope": "tenant:acme-corp/workspace:prod",
  "source": "cycles-server",
  "data": {
    "scope": "tenant:acme-corp/workspace:prod",
    "unit": "USD_MICROCENTS",
    "threshold": 0.80,
    "utilization": 0.82,
    "allocated": 100000000,
    "remaining": 18000000,
    "spent": 67000000,
    "reserved": 15000000,
    "direction": "rising"
  }
}

budget.over_limit_entered

Emitted when debt exceeds overdraft_limit.

{
  "event_id": "evt_aabbccdd11223344",
  "event_type": "budget.over_limit_entered",
  "category": "budget",
  "timestamp": "2026-04-01T14:45:12.345Z",
  "tenant_id": "acme-corp",
  "scope": "tenant:acme-corp/workspace:prod",
  "source": "cycles-server",
  "data": {
    "scope": "tenant:acme-corp/workspace:prod",
    "unit": "USD_MICROCENTS",
    "debt": 15000000,
    "overdraft_limit": 10000000,
    "is_over_limit": true,
    "debt_utilization": 1.5
  }
}

tenant.suspended

Emitted when a tenant is suspended.

{
  "event_id": "evt_5566778899aabbcc",
  "event_type": "tenant.suspended",
  "category": "tenant",
  "timestamp": "2026-04-01T09:00:00.000Z",
  "tenant_id": "acme-corp",
  "source": "cycles-admin",
  "actor": {
    "type": "admin"
  },
  "data": {
    "tenant_id": "acme-corp",
    "new_status": "SUSPENDED",
    "changed_fields": ["status"]
  }
}

api_key.auth_failed

Emitted when authentication fails (invalid or revoked key).

{
  "event_id": "evt_ddee0011ff223344",
  "event_type": "api_key.auth_failed",
  "category": "api_key",
  "timestamp": "2026-04-01T11:22:33.456Z",
  "tenant_id": "acme-corp",
  "source": "cycles-admin",
  "data": {
    "key_id": "key_expired_abc",
    "failure_reason": "KEY_EXPIRED",
    "source_ip": "203.0.113.42"
  }
}

Webhook Delivery Headers

Every webhook POST includes these headers:

POST /your-webhook-endpoint HTTP/1.1
Content-Type: application/json
X-Cycles-Event-Id: evt_a1b2c3d4e5f67890
X-Cycles-Event-Type: reservation.denied
X-Cycles-Signature: sha256=a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2
User-Agent: cycles-server-events/0.1.25.1

Signature Verification

Always verify the X-Cycles-Signature header before processing a webhook:

Python

import hmac
import hashlib

def verify_webhook(body: bytes, secret: str, signature: str) -> bool:
    expected = "sha256=" + hmac.new(
        secret.encode("utf-8"), body, hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, signature)

# In your Flask/FastAPI handler:
@app.post("/webhook")
async def handle_webhook(request: Request):
    body = await request.body()
    sig = request.headers.get("X-Cycles-Signature", "")
    if not verify_webhook(body, SIGNING_SECRET, sig):
        return Response(status_code=401)

    event = json.loads(body)
    event_type = event["event_type"]
    # Route to handler...

Node.js

const crypto = require('crypto');

function verifyWebhook(body, secret, signature) {
  const expected = 'sha256=' + crypto
    .createHmac('sha256', secret)
    .update(body)
    .digest('hex');
  if (expected.length !== signature.length) return false;
  return crypto.timingSafeEqual(
    Buffer.from(expected),
    Buffer.from(signature)
  );
}

// In Express:
app.post('/webhook', express.raw({ type: 'application/json' }), (req, res) => {
  const sig = req.headers['x-cycles-signature'] || '';
  if (!verifyWebhook(req.body, SIGNING_SECRET, sig)) {
    return res.status(401).send('Invalid signature');
  }

  const event = JSON.parse(req.body.toString());
  console.log(`Event: ${event.event_type} for tenant ${event.tenant_id}`);
  res.status(200).json({ received: true });
});

Go

import (
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
)

func verifyWebhook(body []byte, secret, signature string) bool {
    mac := hmac.New(sha256.New, []byte(secret))
    mac.Write(body)
    expected := "sha256=" + hex.EncodeToString(mac.Sum(nil))
    return hmac.Equal([]byte(expected), []byte(signature))
}

Java / Spring Boot

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;

@RestController
public class WebhookController {

    @Value("${cycles.webhook.signing-secret}")
    private String signingSecret;

    @PostMapping("/webhook")
    public ResponseEntity<Void> handleWebhook(
            @RequestBody byte[] body,
            @RequestHeader("X-Cycles-Signature") String signature) {

        if (!verifySignature(body, signingSecret, signature)) {
            return ResponseEntity.status(401).build();
        }

        String json = new String(body, StandardCharsets.UTF_8);
        // Parse and route event...
        return ResponseEntity.ok().build();
    }

    private boolean verifySignature(byte[] body, String secret, String signature) {
        try {
            Mac mac = Mac.getInstance("HmacSHA256");
            mac.init(new SecretKeySpec(
                secret.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
            byte[] hash = mac.doFinal(body);
            StringBuilder hex = new StringBuilder("sha256=");
            for (byte b : hash) hex.append(String.format("%02x", b));
            // Constant-time comparison to prevent timing attacks
            return MessageDigest.isEqual(
                hex.toString().getBytes(StandardCharsets.UTF_8),
                signature.getBytes(StandardCharsets.UTF_8));
        } catch (Exception e) {
            return false;
        }
    }
}

Raw body required

Spring Boot parses JSON by default. Use byte[] as the parameter type (or configure HttpMessageConverter) to get the raw bytes for HMAC verification. If you parse JSON first, whitespace differences will produce a different hash.

Integration: PagerDuty

Route budget alerts to PagerDuty for on-call incident response.

Setup

# Create subscription for budget and security alert events
curl -X POST http://localhost:7979/v1/admin/webhooks \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-middleware.example.com/cycles-to-pagerduty",
    "event_types": [
      "budget.exhausted",
      "budget.over_limit_entered",
      "budget.threshold_crossed",
      "reservation.denied",
      "api_key.auth_failed"
    ],
    "signing_secret": "pd-webhook-secret-abc123",
    "disable_after_failures": 20
  }'

Middleware (Python)

Transform Cycles events into PagerDuty Events API v2 format:

import json
import requests

PAGERDUTY_ROUTING_KEY = "your-pagerduty-integration-key"

SEVERITY_MAP = {
    "budget.exhausted": "critical",
    "budget.over_limit_entered": "critical",
    "budget.threshold_crossed": "warning",
    "reservation.denied": "warning",
    "api_key.auth_failed": "info",
}

@app.post("/cycles-to-pagerduty")
async def forward_to_pagerduty(request: Request):
    body = await request.body()
    # Verify signature first (see above)

    event = json.loads(body)
    severity = SEVERITY_MAP.get(event["event_type"], "info")

    pd_payload = {
        "routing_key": PAGERDUTY_ROUTING_KEY,
        "event_action": "trigger",
        "dedup_key": event["event_id"],  # Correlates retries to the same PD alert
        "payload": {
            "summary": f"[Cycles] {event['event_type']} — tenant: {event['tenant_id']}",
            "severity": severity,
            "source": event.get("scope", event["tenant_id"]),
            "component": event["source"],
            "group": event["category"],
            "custom_details": event.get("data", {})
        }
    }

    requests.post(
        "https://events.pagerduty.com/v2/enqueue",
        json=pd_payload
    )
    return {"ok": True}

What triggers PagerDuty alerts

Cycles Event	PagerDuty Severity	When
budget.exhausted	Critical	Budget remaining = 0, all reservations denied
budget.over_limit_entered	Critical	Debt exceeded overdraft limit; new reservations blocked until debt repaid
budget.threshold_crossed (95%)	Warning	Budget nearly depleted
reservation.denied	Warning	Agent couldn't reserve budget

Integration: Slack

Post budget notifications to a Slack channel.

Setup

# Subscribe to specific budget and tenant alert events
curl -X POST http://localhost:7979/v1/admin/webhooks \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-middleware.example.com/cycles-to-slack",
    "event_types": [
      "budget.threshold_crossed",
      "budget.exhausted",
      "budget.over_limit_entered",
      "budget.funded",
      "reservation.denied",
      "tenant.suspended",
      "tenant.closed"
    ],
    "signing_secret": "slack-webhook-secret-xyz"
  }'

Note: event_categories is additive with event_types. If you specify "event_categories": ["budget"], you receive all budget.* events (15 types including budget.created, budget.debited, etc.), not just the ones in event_types. Use event_types alone when you want precise control over which events trigger notifications.

Middleware (Node.js)

const SLACK_WEBHOOK_URL = 'https://hooks.slack.com/services/T.../B.../xxx';

const EMOJI = {
  'budget.exhausted': ':rotating_light:',
  'budget.over_limit_entered': ':no_entry:',
  'budget.threshold_crossed': ':warning:',
  'budget.funded': ':money_with_wings:',
  'reservation.denied': ':no_entry:',
  'tenant.suspended': ':pause_button:',
  'tenant.closed': ':stop_sign:',
};

// Format amount based on unit type (protocol supports multiple units)
function formatAmount(amount, unit) {
  switch (unit) {
    case 'USD_MICROCENTS': return `$${(amount / 1000000).toFixed(2)}`;
    case 'TOKENS':         return `${amount.toLocaleString()} tokens`;
    case 'CREDITS':        return `${amount.toLocaleString()} credits`;
    case 'RISK_POINTS':    return `${amount.toLocaleString()} risk points`;
    default:               return `${amount.toLocaleString()} ${unit || 'units'}`;
  }
}

app.post('/cycles-to-slack', express.raw({ type: 'application/json' }), async (req, res) => {
  // Verify signature first (see Signature Verification above)

  const event = JSON.parse(req.body.toString());
  const emoji = EMOJI[event.event_type] || ':bell:';
  const data = event.data || {};

  let text = `${emoji} *${event.event_type}*\n`;
  text += `Tenant: \`${event.tenant_id}\`\n`;
  if (event.scope) text += `Scope: \`${event.scope}\`\n`;

  if (data.utilization !== undefined) {
    text += `Utilization: ${(data.utilization * 100).toFixed(1)}%\n`;
  }
  if (data.remaining !== undefined) {
    text += `Remaining: ${formatAmount(data.remaining, data.unit)}\n`;
  }
  if (data.reason_code) {
    text += `Reason: ${data.reason_code}\n`;
  }

  await fetch(SLACK_WEBHOOK_URL, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      text,
      unfurl_links: false,
    }),
  });

  res.status(200).json({ ok: true });
});

Example Slack messages

:warning: budget.threshold_crossed
Tenant: `acme-corp`
Scope: `tenant:acme-corp/workspace:prod`
Utilization: 82.0%
Remaining: $18.00

:rotating_light: budget.exhausted
Tenant: `acme-corp`
Scope: `tenant:acme-corp/workspace:prod`
Utilization: 100.0%
Remaining: $0.00

:no_entry: reservation.denied
Tenant: `acme-corp`
Scope: `tenant:acme-corp/workspace:prod/agent:support-bot`
Reason: BUDGET_EXCEEDED

Integration: ServiceNow

Create incidents in ServiceNow for critical budget events.

Setup

curl -X POST http://localhost:7979/v1/admin/webhooks \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-middleware.example.com/cycles-to-servicenow",
    "event_types": [
      "budget.over_limit_entered",
      "budget.exhausted",
      "system.store_connection_lost"
    ],
    "signing_secret": "snow-secret-123"
  }'

Middleware (Python)

import hmac
import hashlib
import json
import requests

SNOW_INSTANCE = "yourcompany.service-now.com"
SNOW_USER = "cycles-integration"
SNOW_PASS = "..."
SIGNING_SECRET = "snow-secret-123"

PRIORITY_MAP = {
    "budget.over_limit_entered": "2",   # High
    "budget.exhausted": "2",            # High
    "system.store_connection_lost": "1", # Critical
}

@app.post("/cycles-to-servicenow")
async def forward_to_snow(request: Request):
    body = await request.body()
    sig = request.headers.get("X-Cycles-Signature", "")
    expected = "sha256=" + hmac.new(
        SIGNING_SECRET.encode("utf-8"), body, hashlib.sha256
    ).hexdigest()
    if not hmac.compare_digest(expected, sig):
        return Response(status_code=401)

    event = json.loads(body)

    # NOTE: caller_id and assignment_group are reference fields. The values
    # below use display values, which requires sysparm_input_display_value=true.
    # For production, use sys_id values instead (e.g., "caller_id": "6816f79cc0a8016401c5a33be04be441")
    # or configure the API call with the display_value parameter.
    incident = {
        "short_description": f"Cycles: {event['event_type']} — {event['tenant_id']}",
        "description": json.dumps(event, indent=2),
        "urgency": PRIORITY_MAP.get(event["event_type"], "3"),
        "category": "Software",
        "subcategory": "Budget Governance",
        "caller_id": "cycles-system",
        "assignment_group": "Platform Engineering",
        "work_notes": f"Cycles event_id: {event['event_id']}\nCategory: {event['category']}",
    }

    requests.post(
        f"https://{SNOW_INSTANCE}/api/now/table/incident",
        json=incident,
        auth=(SNOW_USER, SNOW_PASS),
        headers={"Content-Type": "application/json"},
        params={"sysparm_input_display_value": "true"}  # Allows display names for reference fields
    )
    return {"ok": True}

Integration: Datadog

Post budget events as Datadog Events for correlation with infrastructure metrics.

Setup

curl -X POST http://localhost:7979/v1/admin/webhooks \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-middleware.example.com/cycles-to-datadog",
    "event_types": [
      "budget.exhausted",
      "budget.over_limit_entered",
      "budget.threshold_crossed",
      "reservation.denied"
    ],
    "signing_secret": "dd-webhook-secret"
  }'

Middleware (Python)

import hmac
import hashlib
import json
import requests

DD_API_KEY = "your-datadog-api-key"
SIGNING_SECRET = "dd-webhook-secret"

ALERT_TYPE_MAP = {
    "budget.exhausted": "error",
    "budget.over_limit_entered": "error",
    "budget.threshold_crossed": "warning",
    "reservation.denied": "warning",
}

@app.post("/cycles-to-datadog")
async def forward_to_datadog(request: Request):
    body = await request.body()
    sig = request.headers.get("X-Cycles-Signature", "")
    expected = "sha256=" + hmac.new(
        SIGNING_SECRET.encode("utf-8"), body, hashlib.sha256
    ).hexdigest()
    if not hmac.compare_digest(expected, sig):
        return Response(status_code=401)

    event = json.loads(body)
    data = event.get("data", {})

    dd_event = {
        "title": f"Cycles: {event['event_type']}",
        "text": f"Tenant: {event['tenant_id']}\nScope: {event.get('scope', 'N/A')}\nSource: {event['source']}",
        "alert_type": ALERT_TYPE_MAP.get(event["event_type"], "info"),
        "source_type_name": "cycles",
        "tags": [
            f"tenant:{event['tenant_id']}",
            f"event_type:{event['event_type']}",
            f"category:{event['category']}",
            f"source:{event['source']}",
        ],
    }

    if data.get("utilization") is not None:
        dd_event["text"] += f"\nUtilization: {data['utilization'] * 100:.1f}%"
    if data.get("scope"):
        dd_event["tags"].append(f"scope:{data['scope']}")

    requests.post(
        "https://api.datadoghq.com/api/v1/events",
        json=dd_event,
        headers={
            "DD-API-KEY": DD_API_KEY,
            "Content-Type": "application/json",
        },
    )
    return {"ok": True}

Event overlays in Datadog

Budget events posted via the Events API appear in Datadog's Events Explorer and can be overlaid on Datadog dashboards. Use tags for filtering — e.g., show only budget.exhausted events on your cost dashboard.

Integration: Microsoft Teams

Post budget alerts to a Teams channel using incoming webhooks.

Setup

curl -X POST http://localhost:7979/v1/admin/webhooks \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-middleware.example.com/cycles-to-teams",
    "event_types": [
      "budget.exhausted",
      "budget.over_limit_entered",
      "budget.threshold_crossed",
      "reservation.denied",
      "tenant.suspended"
    ],
    "signing_secret": "teams-webhook-secret"
  }'

Middleware (Python)

Transform Cycles events into Adaptive Card format for Teams:

Note: Microsoft says Microsoft 365 Connectors are nearing deprecation and recommends the Workflows app going forward. Incoming Webhooks and Adaptive Card posting are still documented and functional, but new development should prefer Power Automate Workflows where possible. The Adaptive Card payload format below works with both approaches.

import hmac
import hashlib
import json
import requests

TEAMS_WEBHOOK_URL = "https://your-org.webhook.office.com/webhookb2/..."  # Legacy connector
# Or Power Automate Workflow HTTP trigger URL
SIGNING_SECRET = "teams-webhook-secret"

CARD_COLOR_MAP = {
    "budget.exhausted": "attention",       # Red
    "budget.over_limit_entered": "attention",
    "budget.threshold_crossed": "warning",  # Yellow
    "reservation.denied": "warning",
    "tenant.suspended": "accent",           # Blue
}

def format_amount(amount, unit):
    if unit == "USD_MICROCENTS":
        return f"${amount / 1_000_000:.2f}"
    return f"{amount:,} {unit.lower()}" if unit else f"{amount:,}"

@app.post("/cycles-to-teams")
async def forward_to_teams(request: Request):
    body = await request.body()
    sig = request.headers.get("X-Cycles-Signature", "")
    expected = "sha256=" + hmac.new(
        SIGNING_SECRET.encode("utf-8"), body, hashlib.sha256
    ).hexdigest()
    if not hmac.compare_digest(expected, sig):
        return Response(status_code=401)

    event = json.loads(body)
    data = event.get("data", {})
    color = CARD_COLOR_MAP.get(event["event_type"], "default")

    facts = [
        {"title": "Tenant", "value": event["tenant_id"]},
        {"title": "Event", "value": event["event_type"]},
        {"title": "Source", "value": event["source"]},
    ]
    if event.get("scope"):
        facts.append({"title": "Scope", "value": event["scope"]})
    if data.get("utilization") is not None:
        facts.append({"title": "Utilization", "value": f"{data['utilization'] * 100:.1f}%"})
    if data.get("remaining") is not None:
        facts.append({"title": "Remaining", "value": format_amount(data["remaining"], data.get("unit"))})
    if data.get("reason_code"):
        facts.append({"title": "Reason", "value": data["reason_code"]})

    card = {
        "type": "message",
        "attachments": [{
            "contentType": "application/vnd.microsoft.card.adaptive",
            "content": {
                "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
                "type": "AdaptiveCard",
                "version": "1.5",
                "body": [
                    {
                        "type": "TextBlock",
                        "size": "medium",
                        "weight": "bolder",
                        "text": f"Cycles: {event['event_type']}",
                        "color": color,
                    },
                    {
                        "type": "FactSet",
                        "facts": facts,
                    },
                ],
            },
        }],
    }

    requests.post(TEAMS_WEBHOOK_URL, json=card)
    return {"ok": True}

Example Teams card

The card renders as a structured fact table:

┌─────────────────────────────────────┐
│ ⚠ Cycles: budget.threshold_crossed │
│                                     │
│ Tenant:      acme-corp              │
│ Event:       budget.threshold_crossed│
│ Source:      cycles-server           │
│ Scope:       tenant:acme-corp/...   │
│ Utilization: 82.0%                  │
│ Remaining:   $18.00                 │
└─────────────────────────────────────┘

Integration: Opsgenie

Route alerts to Opsgenie for on-call management (popular with Atlassian/Jira teams).

Setup

curl -X POST http://localhost:7979/v1/admin/webhooks \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-middleware.example.com/cycles-to-opsgenie",
    "event_types": [
      "budget.exhausted",
      "budget.over_limit_entered",
      "reservation.denied",
      "system.store_connection_lost"
    ],
    "signing_secret": "og-webhook-secret"
  }'

Middleware (Python)

import hmac
import hashlib
import json
import requests

OPSGENIE_API_KEY = "your-opsgenie-api-key"
SIGNING_SECRET = "og-webhook-secret"

PRIORITY_MAP = {
    "budget.exhausted": "P2",
    "budget.over_limit_entered": "P1",
    "system.store_connection_lost": "P1",
    "reservation.denied": "P3",
}

@app.post("/cycles-to-opsgenie")
async def forward_to_opsgenie(request: Request):
    body = await request.body()
    sig = request.headers.get("X-Cycles-Signature", "")
    expected = "sha256=" + hmac.new(
        SIGNING_SECRET.encode("utf-8"), body, hashlib.sha256
    ).hexdigest()
    if not hmac.compare_digest(expected, sig):
        return Response(status_code=401)

    event = json.loads(body)

    alert = {
        "message": f"Cycles: {event['event_type']} — {event['tenant_id']}",
        "alias": event["event_id"],  # Dedup key — same event won't create duplicate alerts
        "description": json.dumps(event, indent=2),
        "priority": PRIORITY_MAP.get(event["event_type"], "P3"),
        "source": event["source"],
        "tags": [event["category"], event["tenant_id"]],
        "entity": event.get("scope", event["tenant_id"]),
        "details": event.get("data", {}),
    }

    requests.post(
        "https://api.opsgenie.com/v2/alerts",
        json=alert,
        headers={
            "Authorization": f"GenieKey {OPSGENIE_API_KEY}",
            "Content-Type": "application/json",
        },
    )
    return {"ok": True}

Note: Opsgenie uses alias for deduplication — setting it to event_id ensures retried webhook deliveries don't create duplicate alerts.

Integration: Custom Receiver (Direct)

For simple use cases, receive webhooks directly without middleware.

Best practices for webhook receivers:

• Acknowledge quickly — return 200 OK as fast as possible. The events service treats non-2xx as failure and will retry with exponential backoff.
• Queue internally — if processing takes time, accept the event, enqueue it in your own job queue, and return 200 immediately.
• Make handlers idempotent — delivery is at-least-once, so you may receive the same event more than once. Use event_id (via X-Cycles-Event-Id header) for deduplication.
• Verify signatures — always check X-Cycles-Signature before processing. Never trust unverified payloads.

from flask import Flask, request
import hmac, hashlib, json

app = Flask(__name__)
SIGNING_SECRET = "your-signing-secret"

@app.post("/webhook")
def handle():
    # 1. Verify signature
    body = request.get_data()
    sig = request.headers.get("X-Cycles-Signature", "")
    expected = "sha256=" + hmac.new(
        SIGNING_SECRET.encode("utf-8"), body, hashlib.sha256
    ).hexdigest()
    if not hmac.compare_digest(expected, sig):
        return "Unauthorized", 401

    # 2. Parse event
    event = json.loads(body)
    event_type = event["event_type"]
    event_id = request.headers.get("X-Cycles-Event-Id")

    # 3. Deduplicate (at-least-once delivery)
    if already_processed(event_id):
        return "OK", 200

    # 4. Route by event type
    if event_type == "budget.exhausted":
        handle_budget_exhausted(event)
    elif event_type == "reservation.denied":
        handle_denial(event)
    elif event_type == "budget.threshold_crossed":
        handle_threshold(event)

    mark_processed(event_id)
    return "OK", 200

Tenant Self-Service Webhooks

Tenants can manage their own webhooks (restricted to budget.*, reservation.*, tenant.* events — 26 of 40 types). Admin-only events (api_key.*, policy.*, system.*) are not available to tenants.

Required API key permissions:

• webhooks:write — create, update, delete subscriptions
• webhooks:read — list subscriptions and delivery history
• events:read — query tenant's event stream via GET /v1/events

If the API key lacks these permissions, the server returns 403 INSUFFICIENT_PERMISSIONS.

# Tenant creates their own webhook using their API key
curl -X POST http://localhost:7979/v1/webhooks \
  -H "X-Cycles-API-Key: $TENANT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://acme-corp.example.com/budget-alerts",
    "event_types": [
      "budget.threshold_crossed",
      "budget.exhausted",
      "reservation.denied"
    ]
  }'

# Response includes the signing_secret (returned ONCE — store it securely):
# {
#   "subscription": { "subscription_id": "whsub_abc123...", ... },
#   "signing_secret": "whsec_dGVzdC1zZWNyZXQ..."
# }

Webhook URL Security

By default, Cycles blocks webhook URLs that resolve to private IP ranges (SSRF protection):

• Blocked by default: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 127.0.0.0/8, 169.254.0.0/16, ::1/128, fc00::/7
• HTTPS required in production. HTTP URLs are rejected unless explicitly enabled.

To test with local endpoints or internal services:

# Enable HTTP and remove CIDR blocks (development only!)
curl -X PUT http://localhost:7979/v1/admin/config/webhook-security \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"allow_http": true, "blocked_cidr_ranges": []}'

For production with internal endpoints, use allowed_url_patterns to allowlist specific internal domains:

curl -X PUT http://localhost:7979/v1/admin/config/webhook-security \
  -H "X-Admin-API-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "allowed_url_patterns": ["https://*.internal.example.com/*"],
    "blocked_cidr_ranges": ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]
  }'

Event Type Reference

Event Type	Produced By	source Field	Use Case
budget.threshold_crossed	Runtime server	cycles-server	Warning: budget nearing limit (default thresholds: 80%, 95%, 100%)
budget.exhausted	Runtime server	cycles-server	Critical: remaining = 0, all reservations denied
budget.over_limit_entered	Runtime server	cycles-server	Critical: debt exceeded overdraft limit; new reservations blocked
budget.over_limit_exited	Admin server	cycles-admin	Recovery: debt repaid below limit
budget.debt_incurred	Runtime server	cycles-server	Info: commit created debt via ALLOW_WITH_OVERDRAFT
reservation.denied	Runtime server	cycles-server	Warning: agent couldn't reserve budget
reservation.commit_overage	Runtime server	cycles-server	Info: actual spend exceeded estimated amount
reservation.expired	Expiry sweeper	expiry-sweeper	Info: reservation TTL expired without commit/release
tenant.suspended	Admin server	cycles-admin	Alert: tenant operations paused
tenant.closed	Admin server	cycles-admin	Alert: tenant permanently closed
api_key.auth_failed	Admin server	cycles-admin	Security: authentication failure
api_key.revoked	Admin server	cycles-admin	Security: key access removed
system.store_connection_lost	Any service	cycles-server	Critical: Redis connection failure
system.webhook_delivery_failed	Events service	cycles-server-events	Meta: webhook delivery permanently failed after all retries

Next steps

• Managing Webhooks — create, test, replay, and monitor webhook subscriptions
• Webhook Event Delivery Protocol — full 40-event-type catalog, delivery headers, retry policy, and status lifecycle
• Deploying the Events Service — deploy the async webhook delivery service
• Security — SSRF protection, signing secret encryption, and deduplication