Overview

An alert is a scheduled check over your span telemetry. Every tool run, play batch, and agent message emits spans, and an alert watches a slice of them: on each tick it computes one metric over the spans since it last ran, compares it to a threshold, and — when the threshold is breached — fires its actions, each as its own run. You define one with defineAlert.

Define an alert

alerts/error-spike.ts

import { defineAlert } from "@cargo-ai/cdk";

import { sentinel } from "../agents/sentinel";
import { slack } from "../connectors/slack";
import { enrich } from "../tools/enrich";

export const errorSpike = defineAlert("error-spike", {
  description: "Error rate of the enrich workflow",
  schedule: { type: "cron", cron: "@every 5m" }, // when the check runs
  scope: { kind: "spans", workflow: enrich }, // which spans it watches
  threshold: { metric: "errorRate", operator: "gte", value: 10 }, // when it breaches
  actions: [
    {
      ref: slack.actions.postMessage,
      config: {
        channelId: "C0123456789",
        format: "markdown",
        body: "Enrich error rate hit {{event.value}}% — {{event.spansUrl}}",
      },
    },
    { ref: sentinel, config: { prompt: "Investigate {{alert.url}}" } },
  ],
});

schedule takes a 5-field cron expression or an @every interval (@every 5m, @every 1h30m), evaluated in UTC. An alert evaluates at most once a minute — every tick scans your spans, so sub-minute intervals are rejected. In the CDK enabled defaults to true, so a deployed alert is armed; set it to false to deploy one without arming it. folder files the alert under an "alert"-kind folder.

Scope: what it watches

The scope’s kind names the data source. Each source pairs with its own metric menu, so a metric can never be asked of a source that can’t compute it.

Spans scope

kind: "spans" watches span telemetry directly. Use any subset of these filters; omitted fields don’t narrow. They mirror the Spans view.

Field	Narrows to
`workflow`	one play or tool’s workflow — a handle, or `workflowRef(uuid)`
`parentAgent`	the spans of runs one agent spawned
`nodeKind`	`native`, `connector`, `tool`, or `agent` nodes
`integration`	one integration slug
`connector`	one connector
`action`	one action slug
`tool`	one tool node
`agent`	one agent node
`executionTitleOrErrorMessage`	spans whose title or error message contains the text (case-insensitive)
`executionStatuses`	`pending`, `success`, and/or `error`
`userUuid`	runs started by one user

parentAgent and agent are different filters. parentAgent matches spans of runs an agent started — the agent-trigger case. agent matches an agent node running inside the watched spans.

workflow takes a play/tool handle or workflowRef(uuid) — not toolRef or agentRef. Spans are keyed by the workflow behind a tool, not by the tool’s own uuid, so passing the wrong kind of reference is rejected at deploy time rather than producing an alert that silently matches nothing.

SQL scope

kind: "orchestrationSql" runs read-only SQL over your orchestration data — the spans, runs, batches, and records tables, automatically scoped to your workspace. The query computes the value itself, so the threshold carries only the comparison:

export const slowNights = defineAlert("slow-nights", {
  schedule: { type: "cron", cron: "0 * * * *" },
  scope: {
    kind: "orchestrationSql",
    sql: `select count(*) from spans
          where execution_status = 'error'
            and execution_started_at > now() - interval 1 hour`,
  },
  threshold: { operator: "gte", value: 50 },
  actions: [
    { ref: sentinel, config: { prompt: "Error spike: {{event.value}}" } },
  ],
});

SQL scopes are not windowed for you — Cargo runs the query exactly as written. Make it self-windowing (as the interval 1 hour above does), otherwise every tick evaluates your whole history.

The value is the first column of the first row, and it must be numeric. A query returning no rows, a NULL, or a non-number produces an error event rather than a value — so an empty aggregate can’t quietly read as 0 and breach an lte threshold.

Threshold: when it breaches

operator is gte (breach at or above value) or lte (breach at or below). For a spans scope, metric says what is measured — it is required, since it decides what the value means. A metric that supports aggregations requires one too:

`metric`	Value	`aggregation`
`errorRate`	failed spans as a percentage (0–100) of the window’s finished spans	—
`duration`	span duration in seconds, over finished spans only	`avg`, `p50`, `p95`, `p99`
`credits`	credits consumed by the window’s spans	`sum`, `avg`, `p95`
`count`	number of spans in the window	—

count with lte is a dead-man’s switch: an empty window really evaluates to 0, so silence breaches. { metric: "count", operator: "lte", value: 0 } tells you a workflow stopped running at all. The other metrics treat an empty window as nothing to judge, not as zero — and for errorRate and duration, “empty” means no finished spans, so a window of runs that are all still going is not judged yet either.

Actions: what fires on breach

Each action becomes its own run, exactly like a play’s healthAlertActions. An action is a connector action, an agent, or a tool, and its config is the input it runs with:

actions: [
  { ref: slack.actions.postMessage, config: { channelId: "C0…", body: "…" } },
  { ref: sentinel, config: { prompt: "…" }, waitUntilFinished: true },
  { ref: enrich, config: {} },
]

An agent’s config is typed already — every agent takes the same { prompt, output? }, so a misspelled key is an editor error in the literal above with nothing to import. A connector action or a tool has an input of its own, and a bare object literal leaves it unchecked. Wrap it in alertConnectorAction / alertToolAction to have TypeScript check config against the real thing:

import { alertConnectorAction, alertToolAction, defineAlert } from "@cargo-ai/cdk";

actions: [
  alertConnectorAction({
    ref: slack.actions.postMessage,
    config: { channelId: "C0…", body: "Error rate {{event.value}}%" },
  }),
  alertToolAction({ ref: enrich, config: { domain: "acme.com" } }),
]

Misspelled and missing fields become editor errors instead of deploy failures, and every field still accepts a {{ … }} template string — so a numeric input can be bound to {{event.value}}. Where the two differ is the source of the schema: a connector action’s comes from cargo-ai cdk types, so an integration you haven’t synced keeps the loose object, while a tool’s comes from its own defineWorkflow input and needs no sync — but a toolRef(uuid) names a tool you didn’t author here, so that one stays loose. Both are helpers rather than the type of actions because TypeScript can’t infer a per-element type through an array literal, the same reason agentConnectorTrigger is one. config is interpolated against the firing under two roots — alert is what you configured, event is what this firing measured — so an action can say what happened:

Variable	Value
`{{alert.name}}` / `{{alert.uuid}}` / `{{alert.url}}`	the alert that fired
`{{event.value}}`	the computed value, rounded to two decimals
`{{event.threshold}}`	the threshold it was compared against
`{{event.operator}}`	`gte` or `lte`
`{{event.windowStart}}` / `{{event.windowEnd}}`	the evaluated window, as ISO timestamps
`{{event.spansUrl}}`	link to the workspace’s Spans view

The evaluation window

An alert does not re-scan a fixed lookback on every tick. Each evaluation covers the time since the previous one, so windows are contiguous and never overlap and every span is judged exactly once:

Window start — where the last evaluation ended. The very first evaluation starts from the moment the alert was last saved.
Window end — slightly behind now, by an allowance for span indexing lag. Spans that land late are picked up by the next tick instead of being missed.

This is why the cron is the window size: @every 5m means each evaluation judges roughly the last five minutes.

Actions fire at most once per window. Before firing, an alert atomically claims its window; if a retry or an overlapping tick already claimed it, nothing is recorded and nothing fires. Actions spawn runs that spend credits and can take real action, so a duplicate is worse than a rare miss — and a sustained breach is detected again on the next tick anyway.

Disabled alerts are skipped, and every evaluation — breach or not — records an event.

From the CLI

cargo-ai observability alert list
cargo-ai observability alert get <alert-uuid>

cargo-ai observability alert create \
  --name "Enrich error rate" \
  --cron "@every 5m" \
  --scope '{"kind":"spans","workflowUuid":"<uuid>"}' \
  --threshold '{"metric":"errorRate","operator":"gte","value":10}'

cargo-ai observability alert update --uuid <alert-uuid> --enabled false
cargo-ai observability alert remove <alert-uuid>

create also takes --actions (a JSON array), --description and --folder. On update, passing none to --description or --folder clears the field, while omitting the flag leaves it untouched.

Unlike defineAlert, an alert created through the API or the CLI starts disabled — arm it with alert update --uuid <uuid> --enabled true once its scope and threshold look right.

Try a scope and threshold before you commit to it — preview computes the value now and reports whether it would breach, without firing anything:

cargo-ai observability alert preview \
  --scope '{"kind":"spans","workflowUuid":"<uuid>"}' \
  --threshold '{"metric":"duration","aggregation":"p95","operator":"gte","value":30}' \
  --window-minutes 60

--window-minutes (default 60) is the lookback for spans scopes; a SQL scope windows itself, so the flag doesn’t affect it.

Using the UI

See Using the UI to build an alert visually, with a live preview of the spans it matches, and Events for reading an alert’s history. An alert built in the UI can be brought back into code: cargo-ai cdk pull writes it as a defineAlert under alerts/, with its scope, threshold and actions referencing the other resources it was pulled alongside.

Get started

Resources

Deploy

CLI

Skills

Reference

Define an alert

Scope: what it watches

Spans scope

SQL scope

Threshold: when it breaches

Actions: what fires on breach

The evaluation window

From the CLI

Using the UI

​Define an alert

​Scope: what it watches

​Spans scope

​SQL scope

​Threshold: when it breaches

​Actions: what fires on breach

​The evaluation window

​From the CLI

​Using the UI

Define an alert

Scope: what it watches

Spans scope

SQL scope

Threshold: when it breaches

Actions: what fires on breach

The evaluation window

From the CLI

Using the UI