Cycles

Appearance

← Blog

· Albert Mavashev · 9 min read

langchainengineeringruntime-authoritybudgetsintegrationsagents
Share Share

Closing the Estimate-Actual Gap with cost_fn ​

Update — 2026-08-06

langchain-runcycles 0.4.0 keeps the cost_fn contract described in this historical release post and adds two operational pieces: cache-aware optional rates (cached_prompt_per_million_usd, cache_read_per_million_usd, and separate 5-minute/1-hour cache-creation rates) and the Python SDK's heartbeat plus durable settlement lifecycle. A failed first commit is journaled before it is surfaced; restart replay and expired-reservation /v1/events recovery keep known spend from disappearing. The rates remain caller-supplied flat pricing, not a live provider-price lookup.

When langchain-runcycles 0.1.5 shipped CyclesModelGate on 2026-05-10, the release notes called out the known limitation directly:

Commits at the configured estimate, not actual token cost.

That is one of the easiest silent bugs to ship in reserve-commit middleware.

The reserve path is correct: authorize before the model call, pre-debit the budget, block if the tenant is out of room. But if the commit path blindly settles at the configured estimate, the ledger stops tracking what the model actually used — even when the provider just reported the exact token counts on the way out, ready to be converted into a real cost number.

langchain-runcycles 0.2.0 closes that gap with cost_fn. This post is about why the gap exists at all, what closing it looks like in AgentMiddleware, and why a separate v0.2.3 correctness patch on settlement HTTP failures matters for the same reason.

Why "commit at estimate" is the default, and the problem with it ​

The reserve-commit lifecycle has an unavoidable asymmetry: at reserve time the action has not happened yet, so the cost is unknown; the reservation has to be sized by an estimate. At commit time the action has happened, so the actual cost is in hand — except that the cleanest default is "use the same estimate we reserved with, because the middleware has no opinion about how to read provider response shapes."

The default works. It also creates a structural bias: every commit is sized to the worst case the estimate covered, regardless of what the model actually billed. Budgets get consumed faster than the workload actually costs. The downstream consequences are exactly the ones described in Estimate Drift: The Silent Killer of Budget Enforcement — once the estimate-to-actual ratio drifts, enforcement thresholds stop tracking real spend.

The fix is structural: commit at actual, not at estimate, whenever the provider gives you usage to read.

How CyclesModelGate commits actual cost: the cost_fn callback ​

The middleware needs a callback that runs after the wrapped model call returns but before the commit goes out, takes whatever the model returned, and produces an Amount to commit. That is cost_fn — exported on the package root as the CostFn type alias for user-supplied extractors:

python
from langchain_runcycles import CostFn  # alias for Callable[[Any], Amount] — (model_response) -> Amount

When CyclesModelGate is constructed with a cost_fn, the middleware calls it after the wrapped model handler returns and uses the returned Amount for commit_reservation instead of the configured estimate. When cost_fn is None (the default), behavior is identical to 0.1.x — the estimate gets committed.

The integration looks like this:

python
from langchain.agents import create_agent
from langchain_runcycles import CyclesModelGate
from langchain_runcycles.extractors import openai_cost
from runcycles import Action, Amount, CyclesClient, CyclesConfig, Subject, Unit

client = CyclesClient(CyclesConfig(base_url="http://localhost:7878", api_key="..."))

model_gate = CyclesModelGate(
    client,
    subject=Subject(tenant="acme", agent="researcher"),
    action=Action(kind="llm.completion", name="gpt-4o"),
    mode="reserve",
    estimate=Amount(unit=Unit.USD_MICROCENTS, amount=2_000_000),  # $0.02 worst case
    cost_fn=openai_cost(
        prompt_per_million_usd=2.50,
        cached_prompt_per_million_usd=1.25,
        completion_per_million_usd=10.00,
    ),
)

agent = create_agent(model="gpt-4o", tools=[...], middleware=[model_gate])

The pre-call reservation still books $0.02 (the estimate). After the model returns, openai_cost reads AIMessage.usage_metadata, multiplies tokens by the configured rates, and produces an Amount. The commit goes out at that real number — typically below the worst-case estimate, sometimes above it when the prompt or response runs unusually long.

If the actual exceeds the reserved estimate, normal Cycles commit-overage policy applies — cost_fn is not a way to bypass the pre-call envelope, just a way to settle the reservation with the provider-reported number. The reserve still has to be sized large enough to cover the call's realistic worst case.

The extractors module ​

Writing a cost_fn from scratch every time would invite the same off-by-10x unit-conversion error that hit the cycles-spring-ai-starter 0.3.0 docs. langchain-runcycles 0.2.0 ships two factory functions:

python
from langchain_runcycles.extractors import openai_cost, anthropic_cost

# Factory kwargs follow each provider's historical pricing vocabulary.
# Both extractors read LangChain's normalized usage_metadata fields
# (`input_tokens` / `output_tokens`) under the hood.
openai = openai_cost(
    prompt_per_million_usd=2.50,
    cached_prompt_per_million_usd=1.25,
    completion_per_million_usd=10.00,
)

anthropic = anthropic_cost(
    input_per_million_usd=3.00,
    output_per_million_usd=15.00,
    cache_read_per_million_usd=0.30,
    cache_creation_5m_per_million_usd=3.75,
    cache_creation_1h_per_million_usd=6.00,
)

Both factories use keyword-only pricing args. That is a deliberate choice — openai_cost(2.50, 10.00) would TypeError, which is exactly the kind of error a developer wants at construction time rather than after a quarter of skewed accounting. The asymmetry between input/prompt cost and output/completion cost is real and persistent; the API surface should not let a caller accidentally swap them. The OpenAI factory uses prompt / completion and the Anthropic factory uses input / output to match each vendor's historical pricing vocabulary; under the hood, both extractors read the same normalized LangChain usage_metadata fields (input_tokens / output_tokens), so the kwarg naming is purely a developer-facing affordance.

Both extractors return Amount in USD_MICROCENTS so the commit path doesn't need a unit translation. Cache counts come from normalized input_token_details; Anthropic's ephemeral_5m_input_tokens and ephemeral_1h_input_tokens can carry different cache-write rates. Omitted tier rates fall back to the generic cache-creation rate and then ordinary input pricing for backward compatibility. Verify the rates and cache tier for the exact model you deploy. For provider-specific tokenizers or custom pricing, write your own cost_fn — the contract is just a callable that receives the model response and returns an Amount (the exported CostFn alias is Callable[[Any], Amount]).

When cost_fn fails ​

Anything that runs between the model returning and the agent receiving the result is a potential way to break the agent. The release notes for 0.2.0 are explicit about this: "cost_fn errors never erase the model result." When cost_fn(result) raises or returns a non-Amount, CyclesModelGate logs a warning and falls back to the configured estimate for the commit. The model result is still returned to the agent. Locked down by tests/test_model_gate.py::test_cost_fn_exception_falls_back_to_estimate, ::test_cost_fn_invalid_return_falls_back_to_estimate, and the async siblings.

That fallback only catches structural failures — an exception or an invalid return type. A cost_fn that runs to completion and returns a perfectly valid Amount based on stale provider pricing will not be caught. The extractor produces a wrong number, the commit lands at the wrong number, and the budget side drifts silently. This is the same drift covered in Estimate Drift: The Silent Killer of Budget Enforcement, just at a different layer. The operational implication is that pricing functions should be versioned, tested against the provider's published pricing page, and treated as policy configuration — not throwaway glue code. The middleware fallback is structural insurance, not a substitute for keeping the rates current.

v0.2.3: failed commits must not look successful ​

The 0.2.0 cost_fn work is half the actuals story. The other half is whether commit and release calls actually report their outcomes — and through v0.2.2, they did not. The runcycles SDK returns CyclesResponse.http_error(...) on HTTP failures without raising. The middleware in 0.2.0–0.2.2 only caught raised exceptions, so a failed commit silently looked like a successful commit, and the documented settlement_error_policy contract was bypassed.

v0.2.3 fixes that on both CyclesModelGate and CyclesToolGate, sync and async paths. Commit HTTP failures now surface via the same settlement_error_policy as exceptions:

  • settlement_error_policy="raise" (default) → RuntimeError carrying denial_reason(response).
  • settlement_error_policy="log" → warning logged, handler result preserved.

Release HTTP failures (best-effort by design) log a warning and never raise. Log message wording also changed from "commit failed" to "commit raised" vs "commit returned HTTP failure" so operators can distinguish the two failure modes in audit logs.

This is a small patch in line count and a meaningful patch in semantics. Reserve-at-estimate / commit-at-actual is structurally honest only if the commit outcome is itself reported honestly. A middleware that silently treats a failed commit as success would let cost_fn produce the correct actual amount and then discard it — the budget side would never learn what the model actually cost.

Version 0.4.0 closes the durability gap that still existed after v0.2.3. Both policies now persist the exact known-spend settlement through the SDK before the first commit attempt. "raise" surfaces CyclesProtocolError after recovery is queued; "log" returns the handler result while the same durable recovery continues. The reservation is heartbeated during long handlers, and an expired commit is recovered through POST /v1/events. The policy is now an observation choice, not a choice between strict and best-effort accounting.

The pattern, generalized ​

Two things have to be true for "commit at actual" to be operationally useful:

  1. Reserve sizes a worst-case envelope; commit lands the real number. The asymmetry between the two times is structural, not accidental. Any middleware that pre-debits a budget around an action with unknown cost has to size by estimate. Any middleware that has access to the action's actual cost afterward should commit by actual.

  2. Settlement outcomes must be observable, not silently absorbed. A successful-looking commit that didn't actually land is worse than a clearly-failed commit — the first lies to the budget; the second triggers operator response.

The same shape is what cycles-spring-ai-starter 0.2.0 did for the Spring AI advisor chain: reserve with the configured estimate, commit on ChatResponse.Usage when the provider populates it, fall back to estimate when it does not. The LangChain story differs in shape — AgentMiddleware is a hook lifecycle, not an advisor chain — but the asymmetry is identical and the resolution is the same.

The contrast worth being honest about is with langchain-runcycles' sibling post on LangGraph. That post is about graph-level controls — per-run, per-node, durable-execution retries, fan-out across sub-graphs. This post is about the middleware-level cost-actuals problem that sits one layer below the graph. Both layers compose; neither replaces the other. The LangGraph piece is about where the gate runs; this post is about what number it commits.

Tool-side actual cost in v0.3.0 ​

The v0.2.x line completed its original scope: model-side cost_fn shipped in v0.2.0, streaming-path verification and regression tests shipped in v0.2.1, the multi-tenant fan-out and HITL demo shipped in v0.2.2, and the settlement-honesty fix shipped in v0.2.3.

Version 0.3.0, released on 2026-05-15, adds cost_fn to CyclesToolGate. The optional callback receives (ToolCallRequest, result) and returns the Amount to commit after the wrapped tool completes. When no callback is supplied—or when it raises or returns an invalid value—the gate preserves the tool result and falls back to committing the configured estimate. Tool result shapes and provider pricing are not normalized, so applications still supply the extractor.

Closing ​

The estimate-as-actual gap is one of the easiest silent failure modes to ship in reserve-commit middleware, and it is fixable with the same shape across frameworks: a callback that reads the provider's reported usage and produces an Amount for commit. cost_fn is the LangChain implementation. The same pattern lives behind cycles-spring-ai-starter's Usage extraction and behind any runtime authority implementation that aims to bill agents at what they actually cost rather than what the worst-case estimate covered.

The release-by-release sequence is worth recording for the trail it leaves. 0.1.5 shipped with the limitation explicit in the release notes. 0.2.0 added model-side cost_fn; 0.2.3 fixed a separate silent-success bug in settlement reporting; and 0.3.0 added the corresponding tool-side callback. Each release moved one gap from known-limitation status to closed, with the contract documented in tests on the way out.

Further reading ​

External references ​

Last updated: